+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20211003/B1b --output-run DEEP/20211003/B1b/science#step1/20240318T051950Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 5.3696 seconds; current memory usage: 0.181 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0063 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0160 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/pipeline_orig.qgraph DEEP/20211003/B1b/science#step1/20240318T051950Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.3275 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 14.1864 seconds; current memory usage: 0.370 Gibyte, delta: 0.189 Gibyte, peak delta: 0.178 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.370 Gibyte (main), 0.355 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0207 seconds; current memory usage: 0.370 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.370 Gibyte (main), 0.355 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 744 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20211003_B1b_science#step1_20240318T051950Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1532 seconds; current memory usage: 0.370 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.370 Gibyte (main), 0.355 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 746 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2385 seconds; current memory usage: 0.372 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.372 Gibyte (main), 0.355 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 8e13877c-feb5-4fa0-8209-fcbd57da8b48 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-18 05:20:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-18 05:20:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:20:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-18 05:20:38 proc_lsst.multi:170 [INFO] [multi] job_id 56975 proc_lsst.multi INFO: [multi] job_id 56975 2024-03-18 05:20:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:38 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 56975 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 56975 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 56975 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 3969ab7b53e8b460d670246c0101f736 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/4d17e71f-ab77-4c8a-b86f-b004797c6b97_isr_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/4d17e71f-ab77-4c8a-b86f-b004797c6b97_isr_1039909_36.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 58b9faaf1090ddc51204969450f808ae parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9f6db3f6-2c9e-403b-9086-081306c695fd_isr_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9f6db3f6-2c9e-403b-9086-081306c695fd_isr_1039907_7.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 821ae211feec246302b06bf56918caab parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d20046ce-be89-432b-88fd-2c5da7884e73_isr_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d20046ce-be89-432b-88fd-2c5da7884e73_isr_1039909_56.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash ea34e850d1f66d03ede4d1c9c3c103a9 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f79611db-466a-4418-9838-6c8a639d6f01_isr_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f79611db-466a-4418-9838-6c8a639d6f01_isr_1039907_12.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash fb0dc3328fa65aee70d12e58f3fdb6ba parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3c488229-5cdf-4f3f-9e61-e8e6f3c495f8_isr_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3c488229-5cdf-4f3f-9e61-e8e6f3c495f8_isr_1040013_8.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash c8d907d3d6a02df4c6b6b70b60fb8697 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/59308bae-1143-4a41-a7f6-e284223d65ee_isr_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/59308bae-1143-4a41-a7f6-e284223d65ee_isr_1039909_43.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 66b03dd02e8efcdc5c6cf11883980f00 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d6365ac7-ddd8-433f-a670-d5b748d25de1_isr_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d6365ac7-ddd8-433f-a670-d5b748d25de1_isr_1039907_33.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash a029ee1209692a8280a6bb38b900bf1a parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/aa135a62-2129-41c7-bc61-77b43ab921c2_isr_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/aa135a62-2129-41c7-bc61-77b43ab921c2_isr_1039908_14.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 26f01fb4327421307263e2ea4a97ea61 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/758125a7-7ce4-41a5-926a-60e8cab0e2e9_isr_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/758125a7-7ce4-41a5-926a-60e8cab0e2e9_isr_1039908_41.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 363db0d5041c376ef61b3181308d33b4 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/161ec3c7-5485-4cad-ab95-424e6a402ac1_isr_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/161ec3c7-5485-4cad-ab95-424e6a402ac1_isr_1039907_41.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash cc5ac018b0d80303a3302bec352776ed parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b1569d44-daf6-49f2-aca2-7009716a7074_isr_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b1569d44-daf6-49f2-aca2-7009716a7074_isr_1039909_29.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 02953d47f5df37791ee27d752b088a18 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/04e27e92-4b79-419a-9bd7-800bc7773b6f_isr_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/04e27e92-4b79-419a-9bd7-800bc7773b6f_isr_1040013_11.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash f79a23370568aa08f3cf06fe3a2b1c43 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2be880e-3576-4247-9bb2-f1f8ced31dd1_isr_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2be880e-3576-4247-9bb2-f1f8ced31dd1_isr_1039907_8.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 3b7cfff8553778a5b6c339efefa4cf66 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/27ddb09d-a91d-4568-ace4-aa56520becfe_isr_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/27ddb09d-a91d-4568-ace4-aa56520becfe_isr_1039908_36.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 112ec71a1b3830686528cbc7afd6cec9 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9d80951b-6e8e-4207-8fa6-e343efa757ea_isr_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9d80951b-6e8e-4207-8fa6-e343efa757ea_isr_1039909_37.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 61196d19c2ec9fd977278b8f164fc3f5 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3a6f466e-fd2d-4d95-bdba-83dc25407bb9_isr_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3a6f466e-fd2d-4d95-bdba-83dc25407bb9_isr_1039909_40.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 7dcfddc7c1ae0fe7985ad1cc8e827d76 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/83a2d955-7b18-4aca-a1b9-d77e3ddf3a06_isr_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/83a2d955-7b18-4aca-a1b9-d77e3ddf3a06_isr_1039908_5.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash b675c7dbc7616b61c96f13d054723c03 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4ad97c2c-edb1-4d78-bf3c-24333a3ffcb3_isr_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4ad97c2c-edb1-4d78-bf3c-24333a3ffcb3_isr_1039908_27.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash f7b70dfd5c347cb70bac63f2b59c0ff1 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81684a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/54e8ac67-0214-48bc-bf02-825eca57ba9a_isr_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/54e8ac67-0214-48bc-bf02-825eca57ba9a_isr_1039909_54.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash ab550904b2c7e2d5c53d6a6c2a8491a2 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/56abe176-d7de-48a6-b59d-3aadf4c65351_isr_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/56abe176-d7de-48a6-b59d-3aadf4c65351_isr_1039909_39.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4d07e2f00963421575be01175d843ea3 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1f5dbdeb-532d-400b-a47e-b12c295da03d_isr_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1f5dbdeb-532d-400b-a47e-b12c295da03d_isr_1039909_27.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 9ff8aa063ab924f5efd8c590aa5b700a parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b2984339-8fa0-4100-b874-c78782bcf073_isr_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b2984339-8fa0-4100-b874-c78782bcf073_isr_1039908_47.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 7410b7dce61cfc405df5aaf22efff657 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cb2a59ad-66ed-4347-8584-9210804c0641_isr_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cb2a59ad-66ed-4347-8584-9210804c0641_isr_1039908_56.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 39bc5d3de70b5233538fa458aedca93c parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26ade41f-969b-44bc-a6b3-cd4e8d5e52c8_isr_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26ade41f-969b-44bc-a6b3-cd4e8d5e52c8_isr_1039908_33.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 502b9cedc2bf5fb29e9cc7064e273c4e parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e760bb92-0f58-481c-81e1-7424c7f26fa3_isr_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e760bb92-0f58-481c-81e1-7424c7f26fa3_isr_1040013_46.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 4bf85b326c5111ea780b3a826582aaeb parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/0eb58d83-5bc7-45e7-ace7-9976fd1b1181_isr_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/0eb58d83-5bc7-45e7-ace7-9976fd1b1181_isr_1039907_1.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash a2779ecd884c1711ba25743ba9d355a9 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/6d50b679-ac19-429a-a6f4-4871b95393db_isr_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/6d50b679-ac19-429a-a6f4-4871b95393db_isr_1039909_16.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash bb0a795e559537ef49d33df143af9ef2 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/03536243-73a2-4366-bbc9-41d382444d9f_isr_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/03536243-73a2-4366-bbc9-41d382444d9f_isr_1039908_17.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 97d0b4dba475507fc54d6fcfa6e94c89 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/152ee5dd-2482-4188-bd93-cbed7f5d4c92_isr_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/152ee5dd-2482-4188-bd93-cbed7f5d4c92_isr_1039909_9.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 1872520bd04684e68f8c512988233d25 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/37970f03-522e-416b-9b5d-4c6d478a9584_isr_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/37970f03-522e-416b-9b5d-4c6d478a9584_isr_1040013_58.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash b95687bdbedfc7003785375059ff36f9 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4f920359-ce1d-4351-af5e-ea178539f654_isr_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4f920359-ce1d-4351-af5e-ea178539f654_isr_1039908_35.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 359a7ef0bbbb045806e8a621aa04aa5b parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6cb98973-8699-409c-a2ee-dfa446e99bf5_isr_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6cb98973-8699-409c-a2ee-dfa446e99bf5_isr_1040013_41.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 8625f839d244e2ff39b0e812884d25fa parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/be48b504-a2c0-4f41-8a08-fe5d8549a6be_isr_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/be48b504-a2c0-4f41-8a08-fe5d8549a6be_isr_1040013_40.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 0b3004b0c87b425c751901d99c1f1ec3 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6bff1df8-6b74-448d-ae9f-6d8ec60a9472_isr_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6bff1df8-6b74-448d-ae9f-6d8ec60a9472_isr_1039908_16.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 2a4cf0d963d0cb0b0d36b1e7c5a71076 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/12cca612-d5f3-4190-9ab3-4bf9b23cac93_isr_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/12cca612-d5f3-4190-9ab3-4bf9b23cac93_isr_1040013_17.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash f175a68a3e8a10ca0367ce2f8bde0286 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/94bcdd35-48f4-4cca-b00e-2960dd059aaf_isr_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/94bcdd35-48f4-4cca-b00e-2960dd059aaf_isr_1039907_49.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 9279dec076d7bbac78c58ea634307531 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/6c0bde1d-7545-4c85-81b5-931a2a8bf4e4_isr_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/6c0bde1d-7545-4c85-81b5-931a2a8bf4e4_isr_1039907_45.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash f6f195e5c7d444c4932a3f046abf66ad parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/a4fef058-19c6-44f5-83ec-5781b7095163_isr_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/a4fef058-19c6-44f5-83ec-5781b7095163_isr_1040013_12.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 291d6c3c0e8d4fd2258902b1b90c95aa parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c921c7db-3a2f-4d3e-ba3a-6077c1e3f710_isr_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c921c7db-3a2f-4d3e-ba3a-6077c1e3f710_isr_1039907_13.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 383bcd171374e1799720724d0355c548 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1c3a22af-5702-4e12-8aff-c3e7c01ef8a2_isr_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1c3a22af-5702-4e12-8aff-c3e7c01ef8a2_isr_1040013_21.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash d6ec24e99284b0668b331849261cf170 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81889a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/365f44db-2da3-4115-b7ef-1475051b2885_isr_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/365f44db-2da3-4115-b7ef-1475051b2885_isr_1040013_61.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 09e4fc22446ec5488ff61cc072841678 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d70320b9-db7c-4762-a43b-2125d8defdfe_isr_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d70320b9-db7c-4762-a43b-2125d8defdfe_isr_1039908_34.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 21277a0aba8118937eacb39acdb0d9e6 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/dda8c6e1-1cb5-4a69-91a7-7df256c555d6_isr_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/dda8c6e1-1cb5-4a69-91a7-7df256c555d6_isr_1039908_18.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 2fec24047422db381026bc4a5a52e221 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9e5e5e87-7e94-4322-be42-535b275bc035_isr_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9e5e5e87-7e94-4322-be42-535b275bc035_isr_1040013_28.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 7c94f1d7cf31a60344460ef0ef030a38 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9f5ea30d-abc3-46fd-9d40-7add1c082964_isr_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9f5ea30d-abc3-46fd-9d40-7add1c082964_isr_1039909_44.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash ad1b0a2553188aab1f1b0fa43bded4f3 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2aadd05f-e9d7-49f9-9b6f-398101e361a9_isr_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2aadd05f-e9d7-49f9-9b6f-398101e361a9_isr_1039907_22.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 07282d6dc2a46d3509dce0ae623b0caa parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/71f10fa4-7693-4143-bd38-5f3511007f59_isr_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/71f10fa4-7693-4143-bd38-5f3511007f59_isr_1039908_54.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash aec079d8b21a6ffe8f246e46cd71f92d parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/634eea31-1221-4765-9b69-517db7781e62_isr_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/634eea31-1221-4765-9b69-517db7781e62_isr_1039908_8.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 172b5a9d3b9827287fdd3b312014686b parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9a84e977-d3e5-4c0d-a17e-e643e9f99117_isr_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9a84e977-d3e5-4c0d-a17e-e643e9f99117_isr_1039908_12.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 3ad7c53130041f1e9a164486ece46d46 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/64b42ac4-8060-4897-a937-7326d7b17ce5_isr_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/64b42ac4-8060-4897-a937-7326d7b17ce5_isr_1039907_54.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 1472177b2c7fe6555046959f45ee03be parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/448414ee-e998-4961-987b-76239b24e69f_isr_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/448414ee-e998-4961-987b-76239b24e69f_isr_1040013_25.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 0c38ebed3df80fb462fadad2546aaa8d parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce3507b4-3d25-472d-9e16-f5004399d026_isr_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce3507b4-3d25-472d-9e16-f5004399d026_isr_1039907_61.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash dd901185394e95c2de56089be853fdbd parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9a6af637-c193-4b01-9e0e-2e814ebc799a_isr_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9a6af637-c193-4b01-9e0e-2e814ebc799a_isr_1039909_30.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash bba6b50c62419c55470d00cce6d5ff73 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/03b94fde-f044-42fd-a3b4-d67cca7849e9_isr_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/03b94fde-f044-42fd-a3b4-d67cca7849e9_isr_1039907_4.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 7fd1d843c934db9a7d90408a3e84a282 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/63b6ddf5-482f-4a3f-bed0-7d79a4054fc8_isr_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/63b6ddf5-482f-4a3f-bed0-7d79a4054fc8_isr_1039908_1.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 8c4ca9f9a83e9da03a84b123e2f3421f parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f0d2827e-1aaf-4fd8-80fb-1ec2b2f172e0_isr_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f0d2827e-1aaf-4fd8-80fb-1ec2b2f172e0_isr_1039907_2.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 6e64026bb1ef5aa35e7219e0fa739b60 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d6a9efa6-4cd2-4fdc-97eb-34402c60e2c8_isr_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d6a9efa6-4cd2-4fdc-97eb-34402c60e2c8_isr_1039909_31.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash c5d07340161e58ebd822977003e07d8d parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f0b0e4e5-b053-4128-95b7-5dd3bb31b067_isr_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f0b0e4e5-b053-4128-95b7-5dd3bb31b067_isr_1039908_15.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash a3ac8105b007715748412ad0f9447dda parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/195d7175-6eda-4478-983f-e8b3845b7623_isr_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/195d7175-6eda-4478-983f-e8b3845b7623_isr_1040013_4.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 82b6677be805c2df9979b43ec505815b parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2bf69f57-252f-4a5d-b769-2e955b3026f6_isr_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2bf69f57-252f-4a5d-b769-2e955b3026f6_isr_1039907_52.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 9ddd56958b79dcba6e44f712ccfd8cc8 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/0c0b16d3-e541-4d30-a7cc-b9425971c879_isr_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/0c0b16d3-e541-4d30-a7cc-b9425971c879_isr_1040013_1.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash d9e3cb503fc8ab0666d3ce6bf85f9a26 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7795da99-66c7-4222-9813-a8b5790e88da_isr_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7795da99-66c7-4222-9813-a8b5790e88da_isr_1040013_51.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash f7ff3ace283180df8bd0b772755f8776 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/36f19dd3-66fc-4415-9c09-06e649280954_isr_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/36f19dd3-66fc-4415-9c09-06e649280954_isr_1039908_11.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash af6666f7c1a135c1f349c5c4cad4be81 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/63a8ba8b-bfa8-4f4d-a83f-c40bb063ff8c_isr_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/63a8ba8b-bfa8-4f4d-a83f-c40bb063ff8c_isr_1039909_7.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash f4bc320a2b86dba8862bf8a6dfcd1c6f parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f2ccf9c5-658b-412a-b6c3-92db5ac0908a_isr_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f2ccf9c5-658b-412a-b6c3-92db5ac0908a_isr_1040013_6.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash bc325acc509b005f3866f06769a1268a parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/60e1cd64-f1b8-4f68-ac8f-5ad8071116c1_isr_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/60e1cd64-f1b8-4f68-ac8f-5ad8071116c1_isr_1039909_33.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 404ac1b7bc6507c9c87659c16f052712 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d619eddc-f5a0-4515-9a65-c20d4b05e941_isr_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d619eddc-f5a0-4515-9a65-c20d4b05e941_isr_1040013_54.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash c01904ec14f31c9404e00a9f8f877d7c parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6624c00a-d10b-4531-9919-c7f29274c884_isr_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6624c00a-d10b-4531-9919-c7f29274c884_isr_1040013_57.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 3b34336e73510aa57787d88436bf2e1a parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/057a6bac-c21a-4fbe-83b8-40d2f7826c37_isr_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/057a6bac-c21a-4fbe-83b8-40d2f7826c37_isr_1039909_24.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 9082cba1d1dbc119b19a1fd9fd79602c parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/49c3563f-b648-48b8-b51c-a03d3ca8545d_isr_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/49c3563f-b648-48b8-b51c-a03d3ca8545d_isr_1039909_62.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 5c1d2ac776194f6897d6fb3442ce5905 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/940149e9-198b-4edd-9bc3-e7869286fb2a_isr_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/940149e9-198b-4edd-9bc3-e7869286fb2a_isr_1040013_2.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash fe02e222b2cc78af286e7c4b44c586c4 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e0e44427-ddc2-42e0-9973-cf4f6dcf5300_isr_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e0e44427-ddc2-42e0-9973-cf4f6dcf5300_isr_1040013_56.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 720ca1041361a056740d3760a99d67ba parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2bd54027-7300-4cb4-895f-c8946e09fc97_isr_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2bd54027-7300-4cb4-895f-c8946e09fc97_isr_1039909_14.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 167d2284e6abaf57977ae16c510d4fc5 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1f97f9a8-bbc3-4b8d-98aa-f2789b226dd2_isr_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1f97f9a8-bbc3-4b8d-98aa-f2789b226dd2_isr_1040013_38.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash c713f851ff3afa1b408d46682a69dc00 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/70c4520c-9a75-4108-a263-accad52c71e8_isr_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/70c4520c-9a75-4108-a263-accad52c71e8_isr_1039907_34.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 3626b63f0aa6ec3f1ee86459e60a8731 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26636b0c-0928-4614-b996-8a4fe773e99c_isr_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26636b0c-0928-4614-b996-8a4fe773e99c_isr_1039908_61.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash d39b3a646005a759522dc0393329c615 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/184f7737-b873-465e-beb7-48cbff9d7f58_isr_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/184f7737-b873-465e-beb7-48cbff9d7f58_isr_1039908_25.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 96853752df4f14add27a0f7d4f4fdb46 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/faba9a61-be97-4b84-80d9-36f4d4301e04_isr_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/faba9a61-be97-4b84-80d9-36f4d4301e04_isr_1039909_47.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 2b400aab721897a13495a95f8be3a846 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/ed391938-d2ba-457d-a60d-42e56ba3fe4d_isr_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/ed391938-d2ba-457d-a60d-42e56ba3fe4d_isr_1039909_19.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 44350b7b255f20519d2963d7e05460a0 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/5f41386f-13aa-44ca-aedc-d9e8e7f6c409_isr_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/5f41386f-13aa-44ca-aedc-d9e8e7f6c409_isr_1039908_55.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash ce83d692bf9d456c6293a4c02374d96e parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/825c2df7-d680-4263-a368-a8d40cbd373c_isr_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/825c2df7-d680-4263-a368-a8d40cbd373c_isr_1039907_35.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 2130215b2077fb8d843d66f9f781bd2e parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/037c55f6-eed3-4b01-be3e-0092820e1272_isr_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/037c55f6-eed3-4b01-be3e-0092820e1272_isr_1039908_29.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 53d009dd13612f4d10bc532fb17f3b17 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bacd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bcedeab2-95d9-4cfd-ad9b-caebc61b826e_isr_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bcedeab2-95d9-4cfd-ad9b-caebc61b826e_isr_1039909_12.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash e3c1f44c8687ec3b425eec32793201a9 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5a55299b-edbc-41b7-8743-95b1914604bf_isr_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5a55299b-edbc-41b7-8743-95b1914604bf_isr_1039907_60.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 9160b292b27a72b51f026599301489ef parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4db7e558-cc04-4526-b9e1-7408ab51b948_isr_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4db7e558-cc04-4526-b9e1-7408ab51b948_isr_1039907_3.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash fd3ebfe2abc1466142d211f90a1d47e5 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2c184267-e1e3-4355-9c3d-544e0680cff3_isr_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2c184267-e1e3-4355-9c3d-544e0680cff3_isr_1040013_50.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash ffa642ad6f99da24989d9c80c9d25d61 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/824cafdf-7d52-4d15-9e47-8091f21401e1_isr_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/824cafdf-7d52-4d15-9e47-8091f21401e1_isr_1039908_31.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 614580f3128c021f29a190311dabaa3e parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6badd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/edbc39e9-2465-4299-9b39-6dfdb3bad5b7_isr_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/edbc39e9-2465-4299-9b39-6dfdb3bad5b7_isr_1039909_26.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 02c951172c851e96ea0af450f4e240cd parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e4527795-6b09-46b5-8d39-04f8ccbc14e2_isr_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e4527795-6b09-46b5-8d39-04f8ccbc14e2_isr_1040013_60.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 4b4654c9f3f3d59d00283968aa4c343d parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ab100783-f33f-4488-aac7-9d03a5b50e59_isr_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ab100783-f33f-4488-aac7-9d03a5b50e59_isr_1040013_18.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 5d5ddae67ba9323207456f123d841472 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b038ba2e-6a31-4447-bcc0-d400c79c7866_isr_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b038ba2e-6a31-4447-bcc0-d400c79c7866_isr_1039908_13.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 999550807bc2c6ee8197d2f5562b9600 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6779707d-df87-4880-9e9b-6f2a77795ed2_isr_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6779707d-df87-4880-9e9b-6f2a77795ed2_isr_1039908_58.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 85e567165538bd5da60c6bffff232737 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2d915c97-f801-4c68-892c-94b816bb47a7_isr_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2d915c97-f801-4c68-892c-94b816bb47a7_isr_1039909_58.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 5fc813be1301e1b17579c4a7fc3f691e parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9f82443f-9923-4d62-a2a9-deeec9e7f7cb_isr_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9f82443f-9923-4d62-a2a9-deeec9e7f7cb_isr_1039908_40.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 8b321d1c9ec4f56d7679aa6c8e7ad00b parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/771bc922-f47c-47c8-8edd-aeae737e7caa_isr_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/771bc922-f47c-47c8-8edd-aeae737e7caa_isr_1040013_44.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 2eab06dac5e554558abc215459e093ea parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/2cc06d3a-b565-4378-b0a4-9ff76adaff43_isr_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/2cc06d3a-b565-4378-b0a4-9ff76adaff43_isr_1039908_9.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 68565531cdd18ec8c851b6d2a3b3a60d parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/08c6e99e-9223-403a-906e-dfc7d7e60141_isr_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/08c6e99e-9223-403a-906e-dfc7d7e60141_isr_1040013_16.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash cdb56036b0475f9db9d0bdbec4cdc2de parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/911930fd-b87f-4380-9fd6-b2cbc2aa46f4_isr_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/911930fd-b87f-4380-9fd6-b2cbc2aa46f4_isr_1039907_28.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 2fa0f04c909470a69d79936e506714aa parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4d4d7f10-074f-44b8-a1b4-18b4b9344546_isr_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4d4d7f10-074f-44b8-a1b4-18b4b9344546_isr_1039907_10.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 44e83b3076218ffc1e8be6ad79e05177 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1065e7c4-ba68-4237-b2e6-c6674530915e_isr_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1065e7c4-ba68-4237-b2e6-c6674530915e_isr_1039909_1.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash b3e85eb7b29392d88a85597126559e6a parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/05f74dc0-46f3-443f-b26c-f8650537a12d_isr_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/05f74dc0-46f3-443f-b26c-f8650537a12d_isr_1039908_3.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash c63ba7cf7446a929db116a0c2b142bd8 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8f77ce2a-82b5-4897-a94c-69463245ce63_isr_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8f77ce2a-82b5-4897-a94c-69463245ce63_isr_1039909_21.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 7f224fb639a23ea2bd2a74ec746ee957 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/79ebfb58-3556-42ff-baad-47cf36ac8329_isr_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/79ebfb58-3556-42ff-baad-47cf36ac8329_isr_1039909_52.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash ece896f6d209840de46a6d24779ddb11 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d38d8cf4-efd6-41fd-9425-3ed14fe55327_isr_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d38d8cf4-efd6-41fd-9425-3ed14fe55327_isr_1039907_57.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 8d6e9b913ff679d5b350492ced088943 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f52f6f0d-c4e8-4fa4-8774-d5292b467603_isr_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f52f6f0d-c4e8-4fa4-8774-d5292b467603_isr_1039907_29.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash a680b4b0bbeb24c9449d5740a8d1ab3a parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/81d0fafd-1932-4c4b-bf41-624e817998f1_isr_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/81d0fafd-1932-4c4b-bf41-624e817998f1_isr_1040013_34.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 16002e13f3eeeaed3eebba4a6d2fc39e parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26694093-ca9d-47d3-a149-251c89d4ecd1_isr_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26694093-ca9d-47d3-a149-251c89d4ecd1_isr_1040013_24.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 0c87066e1b5c05c430bfefc05b2fbd88 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/84045db3-1751-451a-91af-f63d0bf0aa83_isr_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/84045db3-1751-451a-91af-f63d0bf0aa83_isr_1039907_59.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash ba4437cc43d93b0404b2b20fdb734365 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cd6dbfce-bbb9-4cf7-8398-601c3566c38c_isr_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cd6dbfce-bbb9-4cf7-8398-601c3566c38c_isr_1039908_38.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 0a0327caba67197cf9e6324462bc0e3b parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/387f3847-1ec0-4ef7-9804-23fbb2780bbd_isr_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/387f3847-1ec0-4ef7-9804-23fbb2780bbd_isr_1039907_48.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 39b60e78176f43407a752c3323be2006 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/118c894a-2afb-4b4c-8d11-af7ec5a1192a_isr_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/118c894a-2afb-4b4c-8d11-af7ec5a1192a_isr_1039909_2.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 600f6c40b0e8ea4c77db4d051aabed00 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/747d5ed9-6547-445d-aa8e-7cc3fe270089_isr_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/747d5ed9-6547-445d-aa8e-7cc3fe270089_isr_1039907_38.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 3c708571d5b59687a65b7a671dabd9c4 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5c8bbd7a-0085-4b93-a49f-fcd418bb56e3_isr_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5c8bbd7a-0085-4b93-a49f-fcd418bb56e3_isr_1039909_45.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 4f44bfb3165534ffd0c30a04ef683bf8 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5eb1b285-0114-4cb3-adaa-d7d2ffc92de5_isr_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5eb1b285-0114-4cb3-adaa-d7d2ffc92de5_isr_1039909_15.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash fdf3ef4b7ed171beadf4ca909770383a parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/e6767b79-0430-42f0-b71e-bc2a9bb2ebdc_isr_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/e6767b79-0430-42f0-b71e-bc2a9bb2ebdc_isr_1039909_61.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash ba2a6ad21c9e939cecc73114d46947fb parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5afcf24b-c67c-459f-81a8-e1646776068d_isr_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5afcf24b-c67c-459f-81a8-e1646776068d_isr_1039907_31.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash ddc77df4d417217acc9152050d7fd251 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/09aeadc4-adef-4d54-8731-30f7b43778fd_isr_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/09aeadc4-adef-4d54-8731-30f7b43778fd_isr_1039908_49.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 9c76d90434470c8570ab7ab07bf432f4 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/4ece4dc6-38e9-4472-8aa4-ea661c213807_isr_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/4ece4dc6-38e9-4472-8aa4-ea661c213807_isr_1040013_29.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 10dafb8745d4c0cf9fb792dbbc7519fa parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e3f8dd81-1c2d-4608-b649-899010da8b52_isr_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e3f8dd81-1c2d-4608-b649-899010da8b52_isr_1039907_26.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 1af59a1e68278bb2beca159830f25297 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/faec5d9f-fc36-44bf-8cc4-7b1f59b145c0_isr_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/faec5d9f-fc36-44bf-8cc4-7b1f59b145c0_isr_1040013_22.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 16a6e7478171e0c5e538d5282e8394a4 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c084a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/279b9145-5ddc-4fa9-99f5-1f621215aea6_isr_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/279b9145-5ddc-4fa9-99f5-1f621215aea6_isr_1039909_18.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 431f3bf53454a1488d09ff8f1bf7625f parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c087c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3ed729bb-77f8-40d1-9745-7bc1387e9d64_isr_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3ed729bb-77f8-40d1-9745-7bc1387e9d64_isr_1040013_55.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 89ecceaba3728477ec8feda0e7316327 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04ade2c7-f51b-468d-8f72-c25c842cc796_isr_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04ade2c7-f51b-468d-8f72-c25c842cc796_isr_1039909_46.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash cb6b6690bc7c8e6b0c27683dd4fdeb03 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/62ff6f7c-ed0a-46ab-a3c2-b05ccc5c9e8f_isr_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/62ff6f7c-ed0a-46ab-a3c2-b05ccc5c9e8f_isr_1039907_44.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 6793193a8c78be350d3a12d80b993cca parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3bcc0839-adaa-44ee-ac45-3d9190865c83_isr_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3bcc0839-adaa-44ee-ac45-3d9190865c83_isr_1040013_13.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash d86df50c22c5ebafa82f58bdcde76793 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c9375631-82eb-44e4-ad73-a14be13d7cbb_isr_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c9375631-82eb-44e4-ad73-a14be13d7cbb_isr_1039908_24.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 58de5082cca542269d04563770ef93a1 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8b76ba52-3bdb-4c75-8fa7-d449f658a059_isr_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8b76ba52-3bdb-4c75-8fa7-d449f658a059_isr_1040013_15.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 3c40b984da5ad7d8189d44ed951202ea parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a50d0dcb-122c-45e8-b323-ca4a5d82a4bd_isr_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a50d0dcb-122c-45e8-b323-ca4a5d82a4bd_isr_1039907_47.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 3540bf650afb4c708b1e7fdc6477305e parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1dc2f655-6ac6-4251-94bc-4c74aa49e99d_isr_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1dc2f655-6ac6-4251-94bc-4c74aa49e99d_isr_1039909_22.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash bdb88cdb81682d76811377ed0fc1dde1 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b99aa1b6-593e-4d2a-9ded-63a956a9aab0_isr_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b99aa1b6-593e-4d2a-9ded-63a956a9aab0_isr_1039907_51.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 245834af0f6c8e20064dd97623f87f30 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:20:39 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/a40817a8-a89c-4963-bfcf-7c6982f458ef_isr_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/a40817a8-a89c-4963-bfcf-7c6982f458ef_isr_1039909_53.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 889133f07f20b4aa0b6e673075dffc49 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a43d219d-8ada-4461-8bad-c9d895ae7102_isr_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a43d219d-8ada-4461-8bad-c9d895ae7102_isr_1039907_53.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 659edfebbb2b55d301bf182ce94883ce parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/20efc07b-ba4e-473e-bc89-4ddaec1d6a3d_isr_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/20efc07b-ba4e-473e-bc89-4ddaec1d6a3d_isr_1039909_5.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 8428cef0de644e0c84e6610855e2a6f0 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7bf2d1ad-cce4-4c78-8902-0237a4fc806e_isr_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7bf2d1ad-cce4-4c78-8902-0237a4fc806e_isr_1040013_62.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 8e7e41ccb0054c2a5f79b5046b9ceea7 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/184ad2c8-64c8-4a79-9fc3-ef3ddd957080_isr_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/184ad2c8-64c8-4a79-9fc3-ef3ddd957080_isr_1039909_28.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash b38e67d6cb653f2d5a999fd3e478099c parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/598ab02c-80e5-4205-8f88-06de549ea412_isr_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/598ab02c-80e5-4205-8f88-06de549ea412_isr_1040013_52.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 4960714afff7c451ca7134cb95e9d1c1 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/0161cb31-34ff-4f96-979d-3139130927ad_isr_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/0161cb31-34ff-4f96-979d-3139130927ad_isr_1039909_32.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 8c8259550376aee8252751f92d6d2670 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 136 parsl.dataflow.strategy DEBUG: Executor multi has 136 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 33 more blocks parsl.executors.status_handling INFO: Scaling out by 33 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-18 05:20:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-18 05:20:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:20:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3f542260-3b14-48da-bb57-d0823759b580_isr_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3f542260-3b14-48da-bb57-d0823759b580_isr_1039909_10.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 71d68c4edcc5432a0f6f257d1f2a2bcc parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/3ebf76c9-060b-43bd-b026-dd703c77c671_isr_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/3ebf76c9-060b-43bd-b026-dd703c77c671_isr_1039907_6.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 46a0944bc902ef6f3f556ad7f2babed2 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85942555-848a-4360-a607-dea350b28948_isr_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85942555-848a-4360-a607-dea350b28948_isr_1039907_36.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash afdec505e727e85a840c2c0a5965bb3a parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8cc55a36-04b5-4ef6-a1d3-e478e8a81499_isr_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8cc55a36-04b5-4ef6-a1d3-e478e8a81499_isr_1039909_41.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 228845564c636b514c0ac7517ae77fca parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ecad1f25-046d-4d52-9a96-743f6e85a9a3_isr_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ecad1f25-046d-4d52-9a96-743f6e85a9a3_isr_1040013_31.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 1983bca8623b9c4f542cb58fbff7bf67 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/349a1fd9-76f5-4238-b8ec-cfe2db290aa6_isr_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/349a1fd9-76f5-4238-b8ec-cfe2db290aa6_isr_1039908_52.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash e2d7b1d390f47ed4a106d01ee2aa991b parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/576218f5-a218-4693-b59c-1635f92e0fc6_isr_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/576218f5-a218-4693-b59c-1635f92e0fc6_isr_1039909_59.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 57a89a68210a64cdcb46fcc5f672d717 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8e3f130e-523c-41c4-9368-b1f8ca00fe36_isr_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8e3f130e-523c-41c4-9368-b1f8ca00fe36_isr_1039908_28.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 69ad7c2f08b1e1b2e9b5ecd908b295e7 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/cb0e4619-6c60-41d1-8468-8e506b6515c7_isr_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/cb0e4619-6c60-41d1-8468-8e506b6515c7_isr_1039907_15.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash b0cb471ff3a2bae23ca4583e633acfed parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/75bdef96-ee09-4196-81b9-2a2c8c576fb5_isr_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/75bdef96-ee09-4196-81b9-2a2c8c576fb5_isr_1039909_50.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash c2a0fff9b19d82730cbec12d62560645 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/97a5a203-e615-4786-aaad-5effc677902f_isr_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/97a5a203-e615-4786-aaad-5effc677902f_isr_1039908_44.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 0c09584b37029f6f22702cb707242a87 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/80267dec-4152-4b89-b259-bf7393b4f833_isr_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/80267dec-4152-4b89-b259-bf7393b4f833_isr_1040013_30.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 5866f7432f05d4038342f3454f99010f parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c802c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/14494a90-16fa-4b7f-ac9b-e4ebd76ce0a2_isr_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/14494a90-16fa-4b7f-ac9b-e4ebd76ce0a2_isr_1039909_35.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash bcf60dcddb484d4ebeec3a7b828e7971 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c805e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/52544494-67b4-4e06-a34d-e2928c8d4c5d_isr_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/52544494-67b4-4e06-a34d-e2928c8d4c5d_isr_1039907_11.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash d57c28d03a1d2efb1978eb62ecb4cf95 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2255ee3-0b90-4f27-9ec1-a3f3f1de7339_isr_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2255ee3-0b90-4f27-9ec1-a3f3f1de7339_isr_1039907_40.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 9d74e7190c19d50f893db2d2d6adc4ec parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fd0a19-2a66-4e57-bbda-bc4c115e29ed_isr_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fd0a19-2a66-4e57-bbda-bc4c115e29ed_isr_1040013_14.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 61c5f10e3946f8f21d768fbe737f7c2b parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/843cf252-293d-46eb-887d-eb064f1cf53d_isr_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/843cf252-293d-46eb-887d-eb064f1cf53d_isr_1039908_22.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash e592b717e9da7eb37dfd1deeff81032f parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5f15ce7a-ca81-4f49-a46b-317c09673257_isr_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5f15ce7a-ca81-4f49-a46b-317c09673257_isr_1040013_23.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 1da0e2efe0458ac056daef6d07d6e14a parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7dff3c35-b4c6-41c9-9476-911a26ac1652_isr_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7dff3c35-b4c6-41c9-9476-911a26ac1652_isr_1040013_3.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash a8709883f1ab78cb9279851bd5767440 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ee41a8bd-429c-4655-b702-234c0c1bf1e4_isr_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ee41a8bd-429c-4655-b702-234c0c1bf1e4_isr_1039907_25.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 6999be00f960aedf3f3c5027a5eed9b5 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5878d538-50f3-41de-8f60-1f06375f95f7_isr_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5878d538-50f3-41de-8f60-1f06375f95f7_isr_1039907_37.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 89cb5bed119a9b5efde5326363357911 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/49ec6837-36d8-4de8-9a39-f3e3f3ff15b1_isr_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/49ec6837-36d8-4de8-9a39-f3e3f3ff15b1_isr_1040013_9.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash df085e50737540f0731db1f6c06f4d2e parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/a8593458-ed84-42ad-acc2-05b787fc5f7e_isr_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/a8593458-ed84-42ad-acc2-05b787fc5f7e_isr_1039908_42.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 2eae6239d7744b565afbf7726a217ae4 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3853a485-fbba-4abb-993c-223853e4af33_isr_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3853a485-fbba-4abb-993c-223853e4af33_isr_1039909_23.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 9d26569f11af9addbe7988f8eedeb2f5 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c532e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a114faf6-fff9-4669-8272-3bb4519c31ac_isr_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a114faf6-fff9-4669-8272-3bb4519c31ac_isr_1039907_14.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash e838f32b8f5aef23d9370a4ff42f616f parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/7ab18d2b-7bf8-4e21-b6d9-990c89295bfa_isr_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/7ab18d2b-7bf8-4e21-b6d9-990c89295bfa_isr_1039909_51.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash a52af33e407c85e7a18e898b3ca44628 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/407c197f-0535-427b-8512-580e4d0d493a_isr_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/407c197f-0535-427b-8512-580e4d0d493a_isr_1039908_45.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 35549484335cc503268ac9b1411f8f8f parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8c752df7-1ede-4d8a-9696-8bbdd01c28b1_isr_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8c752df7-1ede-4d8a-9696-8bbdd01c28b1_isr_1039907_9.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash e8aa18287cb7e6aca275137cee37bd33 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/bb615016-7014-4dfb-8451-efdf0d148d1d_isr_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/bb615016-7014-4dfb-8451-efdf0d148d1d_isr_1040013_59.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 21346d2410b0deff8f61712576573b35 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c818a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d531925a-9b4b-4c7a-8a6b-7910ce33a925_isr_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d531925a-9b4b-4c7a-8a6b-7910ce33a925_isr_1039907_50.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 4611109f61e31ca1416f638e6c7933d0 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c513cbcb-6785-492c-b364-3d47f068bba6_isr_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c513cbcb-6785-492c-b364-3d47f068bba6_isr_1039908_39.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 7f8f0de5bad3c13f895e7ac204055a8a parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/508285a5-7b61-421f-ab0d-47c6fd09c434_isr_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/508285a5-7b61-421f-ab0d-47c6fd09c434_isr_1039909_3.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 2b5eeccd4b9f404c07952401c5ff642a parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/97631a7e-c8e8-48ae-8ba0-41f2bd9dbfb1_isr_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/97631a7e-c8e8-48ae-8ba0-41f2bd9dbfb1_isr_1039909_57.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash f16f3c50b99893648b614cfddb43ac30 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1e8c9e7b-7e09-4065-8a62-499c25c3f22c_isr_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1e8c9e7b-7e09-4065-8a62-499c25c3f22c_isr_1039909_17.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash a59317917ba91d72da529f2f3c0af349 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d2c98e0c-6ea8-4945-88ac-4e9b468fe300_isr_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d2c98e0c-6ea8-4945-88ac-4e9b468fe300_isr_1040013_43.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash de6a9914e2ea8211cf764793f543cc20 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b7df1e4e-db75-4df8-8098-65fea8ee4d7d_isr_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b7df1e4e-db75-4df8-8098-65fea8ee4d7d_isr_1039907_56.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash fd55bb3c3bf232cdbc12724fa6681850 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/58777bce-5e34-486f-ad9e-4fe0e65ef9dd_isr_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/58777bce-5e34-486f-ad9e-4fe0e65ef9dd_isr_1039909_25.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 21e43d6dcf2521079a9ed5c011ef71ea parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c831a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b58575d6-c1a7-4543-b82e-7e2a6c68b327_isr_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b58575d6-c1a7-4543-b82e-7e2a6c68b327_isr_1039909_42.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 71b18788971d31759cf5ada06e465b01 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c834c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/231559c5-5b1f-40cf-a107-b4a4cfc216f4_isr_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/231559c5-5b1f-40cf-a107-b4a4cfc216f4_isr_1039908_2.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash dd2ba18793033b70b4aee4f5c73d34f3 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c837e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/18d54490-e8ca-44a0-ab57-9788de0d3074_isr_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/18d54490-e8ca-44a0-ab57-9788de0d3074_isr_1039908_50.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 644d3c3bb7ca9dbd03ce87f2fa84421c parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1b828630-e93b-40ce-9e9b-f288a45c7f06_isr_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1b828630-e93b-40ce-9e9b-f288a45c7f06_isr_1040013_33.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash aa1520998ab13186647bd66564e8ff2d parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/0d770d5f-9eed-4523-9db5-7f7936227465_isr_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/0d770d5f-9eed-4523-9db5-7f7936227465_isr_1039908_10.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash e63d6f72e7007824069695ba704b992f parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/e38996f9-d49e-466e-abeb-805aea4a61de_isr_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/e38996f9-d49e-466e-abeb-805aea4a61de_isr_1039908_57.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 9c2a3c8dbe392c79986062ca34ce5756 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:20:39 proc_lsst.multi:170 [INFO] [multi] job_id 17195092 proc_lsst.multi INFO: [multi] job_id 17195092 2024-03-18 05:20:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195092 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195092 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17195092 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-18 05:20:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-18 05:20:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9f830c44-c28a-4c6a-b982-dfbb99cc2215_isr_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9f830c44-c28a-4c6a-b982-dfbb99cc2215_isr_1040013_53.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash f0e06b395f37611bd1a480961d7d811d parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/871dedee-2286-47cd-911a-747241541f0d_isr_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/871dedee-2286-47cd-911a-747241541f0d_isr_1040013_20.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 067b1f96573b8cc5311d383b6bbdfba3 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/07a756d0-14fc-412a-b9ce-3baabead3465_isr_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/07a756d0-14fc-412a-b9ce-3baabead3465_isr_1039907_39.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 05ad21bbc1e6c78244ab6dd12a56d96f parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2080addf-f795-42b8-b4f3-b6b8759d77f4_isr_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2080addf-f795-42b8-b4f3-b6b8759d77f4_isr_1039909_48.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 679c6f2270355370ca6f1d6d6aba1d18 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85605bd4-4e00-4730-84d4-1ea80203341c_isr_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85605bd4-4e00-4730-84d4-1ea80203341c_isr_1039907_58.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 09348c7ac08e42feacb3a6dfc129ca26 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cf0bec3e-e325-41f7-88d3-57799ef4c5dd_isr_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cf0bec3e-e325-41f7-88d3-57799ef4c5dd_isr_1039908_20.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 35ff379b3d5679d64df380b4a4943f5e parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/82ad1849-a1ab-4446-a0ef-e14e22210b7a_isr_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/82ad1849-a1ab-4446-a0ef-e14e22210b7a_isr_1039908_30.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash a1e5c93f4222f5f262d00a50e35ae16a parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9542b44e-67d3-4d40-b8cd-31b1d9377bab_isr_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9542b44e-67d3-4d40-b8cd-31b1d9377bab_isr_1039908_59.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 40ac9a31f173ae14e4fb539bd167b96b parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6addbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c5acdaa1-f12d-4cfe-8aba-704396146dbb_isr_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c5acdaa1-f12d-4cfe-8aba-704396146dbb_isr_1039908_7.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 131ad8949464bd273fab832e21dd4327 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6addee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f1ea8bdd-1d9c-42d3-9c61-21cfb461af08_isr_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f1ea8bdd-1d9c-42d3-9c61-21cfb461af08_isr_1040013_39.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 829de2877f38282e4a113e4f65d7ae93 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b113a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b5f87dab-9dfe-4cd5-9581-e01aefad8715_isr_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b5f87dab-9dfe-4cd5-9581-e01aefad8715_isr_1039909_60.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 1cce2d0d9388fb9e4a400d3005eb3da2 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/717f5ff3-4cee-43c3-963d-fff5d596bfb8_isr_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/717f5ff3-4cee-43c3-963d-fff5d596bfb8_isr_1039909_38.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash c1b10f8c3ea604dd6907363168eb670a parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/eebb55ae-88d3-4af5-bf6e-4e57affc260d_isr_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/eebb55ae-88d3-4af5-bf6e-4e57affc260d_isr_1040013_48.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash ccd0660a19c1bc50b2c4fd5060739ce1 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fc7720-8bf2-4e01-9c78-c0bb679ec5ee_isr_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fc7720-8bf2-4e01-9c78-c0bb679ec5ee_isr_1040013_32.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 05e00f43bec0eae29d18eef63cb18309 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/09d40446-8e50-4e88-98ef-d24582cde1c3_isr_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/09d40446-8e50-4e88-98ef-d24582cde1c3_isr_1039909_13.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 90da23f0bee31fd829ea2cd3c06311c4 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5a64a550-cb67-4801-be76-761aa35785f9_isr_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5a64a550-cb67-4801-be76-761aa35785f9_isr_1040013_36.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash a3c6287585edd9126541a22ff7501409 parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1041e058-b3dd-450e-bb88-7e15a1af15a5_isr_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1041e058-b3dd-450e-bb88-7e15a1af15a5_isr_1039909_6.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 124950e10eac098ee9502cc7df90327c parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3686448c-4bc4-4dc6-83d1-9d4ff1c78fdc_isr_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3686448c-4bc4-4dc6-83d1-9d4ff1c78fdc_isr_1039909_8.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash ac89255fb1532abb0a3cfbab77eae25c parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce616ff0-2fd4-4113-b25d-e4f742d38556_isr_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce616ff0-2fd4-4113-b25d-e4f742d38556_isr_1039907_20.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 4a383590ca19748f66ee89ca1464a8e3 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9354e3ed-0d87-497d-8814-5ecb6c206a1d_isr_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9354e3ed-0d87-497d-8814-5ecb6c206a1d_isr_1039909_4.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 6ed341cfd815f2b05faccb1dcbe8611f parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26b186e0-0652-4a85-8c60-352d8a4441f9_isr_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26b186e0-0652-4a85-8c60-352d8a4441f9_isr_1040013_49.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 38c238499c6096371b0cb51dbfb49a09 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c203b9b1-5713-48f9-aa6c-f0da3f7cc633_isr_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c203b9b1-5713-48f9-aa6c-f0da3f7cc633_isr_1039907_30.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 59ac225f5953aaa32c93b9ff3a5981ae parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f19c3536-0e90-44e5-84e8-d783914596cb_isr_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f19c3536-0e90-44e5-84e8-d783914596cb_isr_1040013_37.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash f699ff9817e678b07dca59484d00e74b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8dc76cec-ce53-4a25-bec7-57d424134cf5_isr_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8dc76cec-ce53-4a25-bec7-57d424134cf5_isr_1039907_32.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash fb4d6b59a74656a58061b80286d5002d parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b5bd4477-cf5e-4532-8ab1-986ff89f5493_isr_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b5bd4477-cf5e-4532-8ab1-986ff89f5493_isr_1039908_37.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash ba7da9bef444b67bc5a4dbf7bf219f24 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6899b143-5cae-4909-ac38-ba047d493def_isr_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6899b143-5cae-4909-ac38-ba047d493def_isr_1040013_27.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 3020fbb5b081ba439d08793ab86b359e parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8f8ac795-e8a5-45f5-a8a6-cbb9d3a01693_isr_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8f8ac795-e8a5-45f5-a8a6-cbb9d3a01693_isr_1039907_43.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 6d37f9dd2f631e472b28ee5fd1ada1d2 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d0e16093-f496-4536-9636-fb2747bca483_isr_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d0e16093-f496-4536-9636-fb2747bca483_isr_1040013_5.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 4cbd0d16e756123382f167dc17940311 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/c1c5a682-cda7-4f09-a7f4-fa92c58b7dee_isr_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/c1c5a682-cda7-4f09-a7f4-fa92c58b7dee_isr_1040013_19.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash db985e69adcf3a7bf3cf514cd286689e parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/1184939a-56b5-447a-9715-8bbbc5e4fa9e_isr_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/1184939a-56b5-447a-9715-8bbbc5e4fa9e_isr_1039907_62.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 5b833df0a43455a886f00a524c336816 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b122a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/49ae7fe2-094b-47d7-9b51-094c9b5adfba_isr_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/49ae7fe2-094b-47d7-9b51-094c9b5adfba_isr_1039908_26.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 87e927fa1f17fbf304eb2917f9e1f8d3 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b125c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d283f1dd-ba1e-4544-9b4e-bf7a0599dc49_isr_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d283f1dd-ba1e-4544-9b4e-bf7a0599dc49_isr_1040013_10.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 0449ba9431f2fbdd9320159c4d02a65f parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b128e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e313ea00-383a-41a3-afb9-07b23113814c_isr_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e313ea00-383a-41a3-afb9-07b23113814c_isr_1039907_21.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 93a34c85483f1700823e1290cbec61cf parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8e2f943a-ce27-4b10-bb15-5fa837e3d604_isr_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8e2f943a-ce27-4b10-bb15-5fa837e3d604_isr_1039907_19.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 676947f6ba52922a4079beb84f5e746b parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/034d3022-adfc-4ee4-8b8d-ababc5e3578c_isr_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/034d3022-adfc-4ee4-8b8d-ababc5e3578c_isr_1039907_27.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 9f3d450cc9e4e5f43288214b8a96fba5 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ef287fef-1a31-408f-b143-2854793ac201_isr_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ef287fef-1a31-408f-b143-2854793ac201_isr_1039907_55.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 7593581bb6410dbadfe5f7e35bba02b4 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6c144fd8-852c-49eb-ade7-a74dbc378754_isr_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6c144fd8-852c-49eb-ade7-a74dbc378754_isr_1039908_51.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 6b06f30a2f37067773e562d6e2414a03 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/73569d78-1f3f-4e1c-a665-47cac99575da_isr_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/73569d78-1f3f-4e1c-a665-47cac99575da_isr_1039908_60.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash ea28ff86cc43c8efc2ce501242932e99 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d4015d94-e462-4138-b6d1-557dac0634ef_isr_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d4015d94-e462-4138-b6d1-557dac0634ef_isr_1039908_62.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 487c7509d3939ab6a7fdd479f20fce6b parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2298c951-f1da-4521-bc62-af7c3b9a05d6_isr_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2298c951-f1da-4521-bc62-af7c3b9a05d6_isr_1040013_42.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash a4102680498de38bd929ad24faa2ff88 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/476b069d-f918-490a-b9ff-7ba4f4c24e89_isr_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/476b069d-f918-490a-b9ff-7ba4f4c24e89_isr_1040013_45.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash d9a1443e5cae22f86450b874f74a55cc parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d10dee52-a152-4d41-a2f2-ebc815a91cd3_isr_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d10dee52-a152-4d41-a2f2-ebc815a91cd3_isr_1039908_48.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 06e43a8715a0217d784d3b8c3502d3e6 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/030a3370-06c8-43d6-a463-a614752b2862_isr_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/030a3370-06c8-43d6-a463-a614752b2862_isr_1039909_49.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash b197b2f752aed9dd2c967ad2412b50e0 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/839f8c6a-44d6-45dc-89df-cec38e2e781b_isr_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/839f8c6a-44d6-45dc-89df-cec38e2e781b_isr_1039908_46.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 7164369d64115f13854e75c47303b30d parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/518f0947-af32-4eaf-8aff-795a01cde991_isr_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/518f0947-af32-4eaf-8aff-795a01cde991_isr_1039907_17.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash c1373a8e115240495a19212790311939 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2c8be529-6ed2-4400-b472-c0527cc01f12_isr_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2c8be529-6ed2-4400-b472-c0527cc01f12_isr_1039909_55.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 7a715d59fdfd13d0c0aa91138cecb978 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8a7a1db5-35d7-4f58-a318-3c00ba113444_isr_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8a7a1db5-35d7-4f58-a318-3c00ba113444_isr_1040013_7.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 661648f09191a32b68646493fed01e16 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bd36580f-60ea-48d6-affc-9a7c0a176048_isr_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bd36580f-60ea-48d6-affc-9a7c0a176048_isr_1039909_20.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 8b1e5f352d7a7f934afe884cc6781feb parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/54ac5339-1b34-46e6-b903-c89e00b91fef_isr_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/54ac5339-1b34-46e6-b903-c89e00b91fef_isr_1039907_42.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 668481be2d4b3cd1ae3c0ce421b57faf parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/56fa741b-3bf5-47e8-9682-5c8e253037e7_isr_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/56fa741b-3bf5-47e8-9682-5c8e253037e7_isr_1040013_47.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash bde3d387619459df45c5415bb4294f24 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d9e1ae79-b442-481a-b873-65275a723e47_isr_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d9e1ae79-b442-481a-b873-65275a723e47_isr_1039908_6.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 437c4c6f397884c6f6ddd242ff658ef0 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5711b65d-944e-4c97-8a0a-4b4eb1c02a86_isr_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5711b65d-944e-4c97-8a0a-4b4eb1c02a86_isr_1039909_34.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash a0f5413695b5cf6928b88f44dabed945 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f5c95187-09f1-4211-b1be-6b13ae59069d_isr_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f5c95187-09f1-4211-b1be-6b13ae59069d_isr_1039908_19.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 8ecb386c9c710c70db7ff26247fce3de parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6a3576e9-11c8-48aa-8830-d0b272b0a9fb_isr_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6a3576e9-11c8-48aa-8830-d0b272b0a9fb_isr_1039908_21.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash cf4e1a8fc00a74e580dde66d445f0e75 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f1df871f-771a-4bd9-9da1-e1d684c3b072_isr_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f1df871f-771a-4bd9-9da1-e1d684c3b072_isr_1039907_24.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash db3187478e2e29ab810f0f9c3a682858 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/b7898236-b20f-41ae-bd8e-1398c527b9de_isr_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/b7898236-b20f-41ae-bd8e-1398c527b9de_isr_1040013_26.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 05058943e33d053567d0e9fa1faf360f parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c23b73e2-909f-45a6-9624-968189f61a1e_isr_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c23b73e2-909f-45a6-9624-968189f61a1e_isr_1039908_53.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 06fc212dbde4a29986a37e130a4437eb parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04d40e86-c3e5-45b5-8196-4150c29ae1d7_isr_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04d40e86-c3e5-45b5-8196-4150c29ae1d7_isr_1039909_11.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 015e63fdf5200fbe9a89a7adf18b39b3 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:20:40 proc_lsst.multi:170 [INFO] [multi] job_id 17195093 proc_lsst.multi INFO: [multi] job_id 17195093 2024-03-18 05:20:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195093 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195093 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17195093 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-18 05:20:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/add0d38f-aca5-4262-86e3-bd58f290e44d_isr_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/add0d38f-aca5-4262-86e3-bd58f290e44d_isr_1039907_5.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 0694186a268277180fd4b43c34a7dfd5 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b64e3f85-afd9-40fe-8ebe-36e823d5bf77_isr_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b64e3f85-afd9-40fe-8ebe-36e823d5bf77_isr_1039907_18.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash a254a83375da60f1f3b37326159d7919 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/eaef1ad0-9940-47b5-8574-705ff73291c0_isr_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/eaef1ad0-9940-47b5-8574-705ff73291c0_isr_1039908_23.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 62e6c864771e823c595a77d11a3f8c54 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/dfbaa47f-093f-4513-b3b5-0f918a659edf_isr_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/dfbaa47f-093f-4513-b3b5-0f918a659edf_isr_1039907_23.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash c5c6b13d8a59364a90e99b09bf33f5dc parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/bb4afa43-1894-4cae-bfa7-34eb1a278211_isr_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/bb4afa43-1894-4cae-bfa7-34eb1a278211_isr_1039908_4.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash a18e702bf10a8e2227dd5617a27b1465 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/cfee3e33-5203-437f-aca2-f8c52f103b8c_isr_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/cfee3e33-5203-437f-aca2-f8c52f103b8c_isr_1040013_35.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash dc839f4152b89f0a69c568d88ac4aaff parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8668169c-7ccf-4f21-82d3-0c7c20ef0e9e_isr_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8668169c-7ccf-4f21-82d3-0c7c20ef0e9e_isr_1039908_32.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash ad12da07f36b6e6d0d59f2ef3b341048 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/834c2b7d-cd23-43c3-8042-8a636b7759b1_isr_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/834c2b7d-cd23-43c3-8042-8a636b7759b1_isr_1039907_16.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 9b7af383bf1d59a352c7f248478dd17a parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c74a4d33-19a0-4448-b89b-592d779a357b_isr_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c74a4d33-19a0-4448-b89b-592d779a357b_isr_1039908_43.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash feb26a18421198d7fa470c0c37225a3b parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9db59b54-40f7-4851-8db4-9f848a92f548_isr_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9db59b54-40f7-4851-8db4-9f848a92f548_isr_1039907_46.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping 2024-03-18 05:20:40 proc_lsst.multi:170 [INFO] [multi] job_id 17195094 proc_lsst.multi INFO: [multi] job_id 17195094 2024-03-18 05:20:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195094 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195094 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17195094 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-18 05:20:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:40 proc_lsst.multi:170 [INFO] [multi] job_id 17195095 proc_lsst.multi INFO: [multi] job_id 17195095 2024-03-18 05:20:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195095 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195095 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17195095 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-18 05:20:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:40 proc_lsst.multi:170 [INFO] [multi] job_id 17195096 proc_lsst.multi INFO: [multi] job_id 17195096 2024-03-18 05:20:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195096 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195096 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17195096 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-18 05:20:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:41 proc_lsst.multi:170 [INFO] [multi] job_id 17195097 proc_lsst.multi INFO: [multi] job_id 17195097 2024-03-18 05:20:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:20:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195097 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195097 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17195097 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-18 05:20:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:41 proc_lsst.multi:170 [INFO] [multi] job_id 17195098 proc_lsst.multi INFO: [multi] job_id 17195098 2024-03-18 05:20:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:20:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195098 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195098 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17195098 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-18 05:20:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:41 proc_lsst.multi:170 [INFO] [multi] job_id 17195099 proc_lsst.multi INFO: [multi] job_id 17195099 2024-03-18 05:20:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:20:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195099 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195099 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17195099 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-18 05:20:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:20:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:42 proc_lsst.multi:170 [INFO] [multi] job_id 17195100 proc_lsst.multi INFO: [multi] job_id 17195100 2024-03-18 05:20:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:20:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195100 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195100 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17195100 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-18 05:20:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:42 proc_lsst.multi:170 [INFO] [multi] job_id 17195101 proc_lsst.multi INFO: [multi] job_id 17195101 2024-03-18 05:20:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:20:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195101 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195101 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17195101 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-18 05:20:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:42 proc_lsst.multi:170 [INFO] [multi] job_id 17195102 proc_lsst.multi INFO: [multi] job_id 17195102 2024-03-18 05:20:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:20:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195102 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195102 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17195102 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-18 05:20:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:20:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:43 proc_lsst.multi:170 [INFO] [multi] job_id 17195103 proc_lsst.multi INFO: [multi] job_id 17195103 2024-03-18 05:20:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:20:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195103 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195103 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17195103 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-18 05:20:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-18 05:20:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:20:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:43 proc_lsst.multi:170 [INFO] [multi] job_id 17195104 proc_lsst.multi INFO: [multi] job_id 17195104 2024-03-18 05:20:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:20:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195104 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195104 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17195104 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-18 05:20:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-18 05:20:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:20:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:44 proc_lsst.multi:170 [INFO] [multi] job_id 17195106 proc_lsst.multi INFO: [multi] job_id 17195106 2024-03-18 05:20:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:20:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195106 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195106 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17195106 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-18 05:20:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:44 proc_lsst.multi:170 [INFO] [multi] job_id 17195107 proc_lsst.multi INFO: [multi] job_id 17195107 2024-03-18 05:20:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:20:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195107 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195107 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17195107 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-18 05:20:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:44 proc_lsst.multi:170 [INFO] [multi] job_id 17195108 proc_lsst.multi INFO: [multi] job_id 17195108 2024-03-18 05:20:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:20:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195108 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195108 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17195108 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-18 05:20:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:20:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195109 proc_lsst.multi INFO: [multi] job_id 17195109 2024-03-18 05:20:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:20:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195109 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195109 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17195109 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-18 05:20:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195110 proc_lsst.multi INFO: [multi] job_id 17195110 2024-03-18 05:20:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:20:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195110 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195110 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17195110 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-18 05:20:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195111 proc_lsst.multi INFO: [multi] job_id 17195111 2024-03-18 05:20:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:20:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195111 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195111 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17195111 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-18 05:20:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195112 proc_lsst.multi INFO: [multi] job_id 17195112 2024-03-18 05:20:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195112 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195112 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17195112 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-18 05:20:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195113 proc_lsst.multi INFO: [multi] job_id 17195113 2024-03-18 05:20:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 05:20:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195113 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195113 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17195113 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-18 05:20:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 05:20:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195114 proc_lsst.multi INFO: [multi] job_id 17195114 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195114 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195114 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17195114 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195115 proc_lsst.multi INFO: [multi] job_id 17195115 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195115 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195115 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17195115 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195116 proc_lsst.multi INFO: [multi] job_id 17195116 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195116 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195116 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17195116 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195117 proc_lsst.multi INFO: [multi] job_id 17195117 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195117 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195117 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17195117 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195118 proc_lsst.multi INFO: [multi] job_id 17195118 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195118 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195118 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17195118 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195119 proc_lsst.multi INFO: [multi] job_id 17195119 2024-03-18 05:20:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 05:20:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195119 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195119 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17195119 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-18 05:20:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 05:20:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195120 proc_lsst.multi INFO: [multi] job_id 17195120 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195120 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195120 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17195120 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195122 proc_lsst.multi INFO: [multi] job_id 17195122 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195122 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195122 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17195122 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195123 proc_lsst.multi INFO: [multi] job_id 17195123 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17195123 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17195123 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17195123 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195124 proc_lsst.multi INFO: [multi] job_id 17195124 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17195124 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17195124 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17195124 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195125 proc_lsst.multi INFO: [multi] job_id 17195125 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17195125 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17195125 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17195125 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195126 proc_lsst.multi INFO: [multi] job_id 17195126 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17195126 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17195126 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17195126 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 34, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/33 running/pending blocks, and 15 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 34 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 28 more blocks parsl.executors.status_handling INFO: Scaling out by 28 blocks parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195127 proc_lsst.multi INFO: [multi] job_id 17195127 2024-03-18 05:20:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17195127 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17195127 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17195127 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-18 05:20:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195128 proc_lsst.multi INFO: [multi] job_id 17195128 2024-03-18 05:20:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17195128 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17195128 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17195128 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-18 05:20:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195130 proc_lsst.multi INFO: [multi] job_id 17195130 2024-03-18 05:20:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195130 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195130 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17195130 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-18 05:20:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195131 proc_lsst.multi INFO: [multi] job_id 17195131 2024-03-18 05:20:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195131 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195131 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17195131 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-18 05:20:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195132 proc_lsst.multi INFO: [multi] job_id 17195132 2024-03-18 05:20:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195132 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195132 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17195132 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-18 05:20:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195133 proc_lsst.multi INFO: [multi] job_id 17195133 2024-03-18 05:20:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195133 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195133 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17195133 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-18 05:20:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195134 proc_lsst.multi INFO: [multi] job_id 17195134 2024-03-18 05:20:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195134 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195134 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17195134 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-18 05:20:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195135 proc_lsst.multi INFO: [multi] job_id 17195135 2024-03-18 05:20:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:20:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195135 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195135 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17195135 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-18 05:20:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195136 proc_lsst.multi INFO: [multi] job_id 17195136 2024-03-18 05:20:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:20:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195136 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195136 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17195136 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-18 05:20:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195137 proc_lsst.multi INFO: [multi] job_id 17195137 2024-03-18 05:20:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:20:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195137 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195137 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17195137 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-18 05:20:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:20:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:50 proc_lsst.multi:170 [INFO] [multi] job_id 17195138 proc_lsst.multi INFO: [multi] job_id 17195138 2024-03-18 05:20:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:20:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195138 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195138 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17195138 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-18 05:20:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:50 proc_lsst.multi:170 [INFO] [multi] job_id 17195139 proc_lsst.multi INFO: [multi] job_id 17195139 2024-03-18 05:20:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:20:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195139 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195139 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17195139 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-18 05:20:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:50 proc_lsst.multi:170 [INFO] [multi] job_id 17195140 proc_lsst.multi INFO: [multi] job_id 17195140 2024-03-18 05:20:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:20:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195140 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195140 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17195140 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-18 05:20:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:50 proc_lsst.multi:170 [INFO] [multi] job_id 17195141 proc_lsst.multi INFO: [multi] job_id 17195141 2024-03-18 05:20:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:20:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195141 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195141 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17195141 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-18 05:20:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:20:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:51 proc_lsst.multi:170 [INFO] [multi] job_id 17195142 proc_lsst.multi INFO: [multi] job_id 17195142 2024-03-18 05:20:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:20:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195142 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195142 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17195142 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-18 05:20:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:51 proc_lsst.multi:170 [INFO] [multi] job_id 17195143 proc_lsst.multi INFO: [multi] job_id 17195143 2024-03-18 05:20:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:20:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195143 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195143 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17195143 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-18 05:20:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:51 proc_lsst.multi:170 [INFO] [multi] job_id 17195144 proc_lsst.multi INFO: [multi] job_id 17195144 2024-03-18 05:20:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:20:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195144 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195144 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17195144 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-18 05:20:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:20:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:52 proc_lsst.multi:170 [INFO] [multi] job_id 17195145 proc_lsst.multi INFO: [multi] job_id 17195145 2024-03-18 05:20:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:20:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195145 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195145 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17195145 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-18 05:20:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:52 proc_lsst.multi:170 [INFO] [multi] job_id 17195146 proc_lsst.multi INFO: [multi] job_id 17195146 2024-03-18 05:20:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:20:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195146 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195146 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17195146 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-18 05:20:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:52 proc_lsst.multi:170 [INFO] [multi] job_id 17195147 proc_lsst.multi INFO: [multi] job_id 17195147 2024-03-18 05:20:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:20:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195147 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195147 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17195147 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-18 05:20:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:20:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:53 proc_lsst.multi:170 [INFO] [multi] job_id 17195148 proc_lsst.multi INFO: [multi] job_id 17195148 2024-03-18 05:20:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:20:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195148 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195148 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17195148 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-18 05:20:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:53 proc_lsst.multi:170 [INFO] [multi] job_id 17195149 proc_lsst.multi INFO: [multi] job_id 17195149 2024-03-18 05:20:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195149 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195149 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17195149 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-18 05:20:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:53 proc_lsst.multi:170 [INFO] [multi] job_id 17195150 proc_lsst.multi INFO: [multi] job_id 17195150 2024-03-18 05:20:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195150 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195150 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17195150 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-18 05:20:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:54 proc_lsst.multi:170 [INFO] [multi] job_id 17195151 proc_lsst.multi INFO: [multi] job_id 17195151 2024-03-18 05:20:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195151 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195151 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17195151 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-18 05:20:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:54 proc_lsst.multi:170 [INFO] [multi] job_id 17195152 proc_lsst.multi INFO: [multi] job_id 17195152 2024-03-18 05:20:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195152 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195152 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17195152 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-18 05:20:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58b9faaf1090ddc51204969450f808ae with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 915d8a8d455d4d590534929d9054b3f6 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:20:54 proc_lsst.multi:170 [INFO] [multi] job_id 17195153 proc_lsst.multi INFO: [multi] job_id 17195153 2024-03-18 05:20:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195153 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195153 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17195153 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-18 05:20:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4610347e-8e58-4aee-8694-496363deff55_characterizeImage_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4610347e-8e58-4aee-8694-496363deff55_characterizeImage_1039907_7.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9f6db3f6-2c9e-403b-9086-081306c695fd_isr_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9f6db3f6-2c9e-403b-9086-081306c695fd_isr_1039907_7.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:54 proc_lsst.multi:170 [INFO] [multi] job_id 17195154 proc_lsst.multi INFO: [multi] job_id 17195154 2024-03-18 05:20:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195154 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195154 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17195154 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-18 05:20:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:20:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:20:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:20:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,10.64.65.9,198.48.92.26,127.0.0.1,10.64.129.9,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54850 --result_port=54634 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:20:55 proc_lsst.multi:170 [INFO] [multi] job_id 17195155 proc_lsst.multi INFO: [multi] job_id 17195155 2024-03-18 05:20:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:20:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:20:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195155 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195155 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17195155 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 45 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3969ab7b53e8b460d670246c0101f736 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 6c78aa95f2f14d2b45573bb61e5f900f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8a9081dc-7485-4a08-84a7-e578d11c0dce_characterizeImage_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8a9081dc-7485-4a08-84a7-e578d11c0dce_characterizeImage_1039909_36.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/4d17e71f-ab77-4c8a-b86f-b004797c6b97_isr_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/4d17e71f-ab77-4c8a-b86f-b004797c6b97_isr_1039909_36.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 56 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8625f839d244e2ff39b0e812884d25fa with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 67c50966fba2768094a351c0c9d017c5 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b6027595-ae34-40e8-a940-1fe6492d65be_characterizeImage_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b6027595-ae34-40e8-a940-1fe6492d65be_characterizeImage_1040013_40.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/be48b504-a2c0-4f41-8a08-fe5d8549a6be_isr_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/be48b504-a2c0-4f41-8a08-fe5d8549a6be_isr_1040013_40.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07282d6dc2a46d3509dce0ae623b0caa with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 69afef8340b0002d8983725f5def3cd5 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/de373069-d837-4c3c-a182-2e1845eab271_characterizeImage_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/de373069-d837-4c3c-a182-2e1845eab271_characterizeImage_1039908_54.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/71f10fa4-7693-4143-bd38-5f3511007f59_isr_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/71f10fa4-7693-4143-bd38-5f3511007f59_isr_1039908_54.stderr 2024-03-18 05:21:10 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:21:10 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:10 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:11 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:11 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:11 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c4ca9f9a83e9da03a84b123e2f3421f with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 194db5cd5efd7c9cb5f3c6c310c6d8f8 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f0d2827e-1aaf-4fd8-80fb-1ec2b2f172e0_isr_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f0d2827e-1aaf-4fd8-80fb-1ec2b2f172e0_isr_1039907_2.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383bcd171374e1799720724d0355c548 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 8c5cc063462a946ab9e75c32ce817e24 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81887c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0379cb32-101d-4862-b825-797092f1e110_characterizeImage_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0379cb32-101d-4862-b825-797092f1e110_characterizeImage_1040013_21.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1c3a22af-5702-4e12-8aff-c3e7c01ef8a2_isr_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1c3a22af-5702-4e12-8aff-c3e7c01ef8a2_isr_1040013_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:12 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4cf0d963d0cb0b0d36b1e7c5a71076 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 175c615926aa84c3e6383171b2b465c5 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/53f84eb9-7f88-4db2-84ce-d72396d88951_characterizeImage_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/53f84eb9-7f88-4db2-84ce-d72396d88951_characterizeImage_1040013_17.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/12cca612-d5f3-4190-9ab3-4bf9b23cac93_isr_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/12cca612-d5f3-4190-9ab3-4bf9b23cac93_isr_1040013_17.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e4fc22446ec5488ff61cc072841678 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash d8323045890a3c4195531e4bf7beea2d parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c61ed480-97f4-4df9-90b1-0699c30e21fc_characterizeImage_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c61ed480-97f4-4df9-90b1-0699c30e21fc_characterizeImage_1039908_34.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d70320b9-db7c-4762-a43b-2125d8defdfe_isr_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d70320b9-db7c-4762-a43b-2125d8defdfe_isr_1039908_34.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9279dec076d7bbac78c58ea634307531 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 00a353909e900d49a6d0991938d50b61 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/3d0c9e30-f20c-4c76-82fa-11b5efc06762_characterizeImage_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/3d0c9e30-f20c-4c76-82fa-11b5efc06762_characterizeImage_1039907_45.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/6c0bde1d-7545-4c85-81b5-931a2a8bf4e4_isr_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/6c0bde1d-7545-4c85-81b5-931a2a8bf4e4_isr_1039907_45.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aec079d8b21a6ffe8f246e46cd71f92d with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 807b40ebca627977e1929e2fa504a86c parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/162ce5bb-4b06-4b53-a617-df4b24f4a4d1_characterizeImage_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/162ce5bb-4b06-4b53-a617-df4b24f4a4d1_characterizeImage_1039908_8.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/634eea31-1221-4765-9b69-517db7781e62_isr_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/634eea31-1221-4765-9b69-517db7781e62_isr_1039908_8.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b3004b0c87b425c751901d99c1f1ec3 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 3cab20add429f3f98728cf8f9bcf798d parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dddb28ac-8b86-48cd-b4b5-c6716646745c_characterizeImage_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dddb28ac-8b86-48cd-b4b5-c6716646745c_characterizeImage_1039908_16.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6bff1df8-6b74-448d-ae9f-6d8ec60a9472_isr_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6bff1df8-6b74-448d-ae9f-6d8ec60a9472_isr_1039908_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:12 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 291d6c3c0e8d4fd2258902b1b90c95aa with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 239af51659c0fd29c7b30f8111e668d8 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/acf09252-27cd-444e-b78e-142dfb93357a_characterizeImage_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/acf09252-27cd-444e-b78e-142dfb93357a_characterizeImage_1039907_13.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c921c7db-3a2f-4d3e-ba3a-6077c1e3f710_isr_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c921c7db-3a2f-4d3e-ba3a-6077c1e3f710_isr_1039907_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:13 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fec24047422db381026bc4a5a52e221 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash eb2666e2910a795c9008e4b4bdbcab11 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3f69c5b0-101f-4efb-bffa-f63c1e84a2d8_characterizeImage_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3f69c5b0-101f-4efb-bffa-f63c1e84a2d8_characterizeImage_1040013_28.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9e5e5e87-7e94-4322-be42-535b275bc035_isr_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9e5e5e87-7e94-4322-be42-535b275bc035_isr_1040013_28.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f175a68a3e8a10ca0367ce2f8bde0286 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 4c8b65a6b4f85225127133e5df24e33f parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b8940c74-6178-4ec8-96ad-ccee02f74370_characterizeImage_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b8940c74-6178-4ec8-96ad-ccee02f74370_characterizeImage_1039907_49.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/94bcdd35-48f4-4cca-b00e-2960dd059aaf_isr_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/94bcdd35-48f4-4cca-b00e-2960dd059aaf_isr_1039907_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:13 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e64026bb1ef5aa35e7219e0fa739b60 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash bc70080b30b15a26284df5810ba0d588 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/17e5ce8d-e7d7-40c3-ad1e-0af57f415f3e_characterizeImage_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/17e5ce8d-e7d7-40c3-ad1e-0af57f415f3e_characterizeImage_1039909_31.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d6a9efa6-4cd2-4fdc-97eb-34402c60e2c8_isr_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d6a9efa6-4cd2-4fdc-97eb-34402c60e2c8_isr_1039909_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:13 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:13 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21277a0aba8118937eacb39acdb0d9e6 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash ac7d1dbf949fc9cfd118813f98f4657b parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/09bfdbb2-4677-4e9b-9c8c-f35be7ab2a51_characterizeImage_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/09bfdbb2-4677-4e9b-9c8c-f35be7ab2a51_characterizeImage_1039908_18.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/dda8c6e1-1cb5-4a69-91a7-7df256c555d6_isr_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/dda8c6e1-1cb5-4a69-91a7-7df256c555d6_isr_1039908_18.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5d07340161e58ebd822977003e07d8d with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 24384de3379c1ca0276e4b233d9507e4 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/33d793df-d729-46dc-9f05-63314a7bcb3d_characterizeImage_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/33d793df-d729-46dc-9f05-63314a7bcb3d_characterizeImage_1039908_15.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f0b0e4e5-b053-4128-95b7-5dd3bb31b067_isr_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f0b0e4e5-b053-4128-95b7-5dd3bb31b067_isr_1039908_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:13 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad1b0a2553188aab1f1b0fa43bded4f3 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 228927625c00ee35a3795d772828c73e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/638e7d0d-f723-4729-a983-aec64227bce3_characterizeImage_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/638e7d0d-f723-4729-a983-aec64227bce3_characterizeImage_1039907_22.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2aadd05f-e9d7-49f9-9b6f-398101e361a9_isr_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2aadd05f-e9d7-49f9-9b6f-398101e361a9_isr_1039907_22.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba6b50c62419c55470d00cce6d5ff73 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 9ab8a1a0eb7b55199f7c31c4b23b2be1 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/892bfcc1-768e-4ded-b208-c03024600cdd_characterizeImage_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/892bfcc1-768e-4ded-b208-c03024600cdd_characterizeImage_1039907_4.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/03b94fde-f044-42fd-a3b4-d67cca7849e9_isr_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/03b94fde-f044-42fd-a3b4-d67cca7849e9_isr_1039907_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:14 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6f195e5c7d444c4932a3f046abf66ad with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 33ec52ff750481e1dbfc29414647fada parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cb380f92-d945-428b-9cba-9adfdfe83bc3_characterizeImage_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cb380f92-d945-428b-9cba-9adfdfe83bc3_characterizeImage_1040013_12.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/a4fef058-19c6-44f5-83ec-5781b7095163_isr_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/a4fef058-19c6-44f5-83ec-5781b7095163_isr_1040013_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:14 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c94f1d7cf31a60344460ef0ef030a38 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash bfbc2ef324735c33f4f1f1d06edb70ca parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0873391f-8951-432b-8425-1d834c095231_characterizeImage_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0873391f-8951-432b-8425-1d834c095231_characterizeImage_1039909_44.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9f5ea30d-abc3-46fd-9d40-7add1c082964_isr_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9f5ea30d-abc3-46fd-9d40-7add1c082964_isr_1039909_44.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fd1d843c934db9a7d90408a3e84a282 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash d16b6281b50395e86b3da64446101c39 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1dd4fcee-0627-4bd8-b25d-13f066914dde_characterizeImage_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1dd4fcee-0627-4bd8-b25d-13f066914dde_characterizeImage_1039908_1.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/63b6ddf5-482f-4a3f-bed0-7d79a4054fc8_isr_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/63b6ddf5-482f-4a3f-bed0-7d79a4054fc8_isr_1039908_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:15 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:15 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:15 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6ec24e99284b0668b331849261cf170 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 8770358a28e9262e067751a9fe9b8247 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/8f470f7f-e7fd-4b6f-9af2-d4f5d640fe80_characterizeImage_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/8f470f7f-e7fd-4b6f-9af2-d4f5d640fe80_characterizeImage_1040013_61.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/365f44db-2da3-4115-b7ef-1475051b2885_isr_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/365f44db-2da3-4115-b7ef-1475051b2885_isr_1040013_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:15 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:15 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:16 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:16 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:16 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02953d47f5df37791ee27d752b088a18 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 6f52073b8badf37326c366da3fb86a11 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f73066d8-5307-472d-8d38-f66e332362d5_characterizeImage_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f73066d8-5307-472d-8d38-f66e332362d5_characterizeImage_1040013_11.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/04e27e92-4b79-419a-9bd7-800bc7773b6f_isr_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/04e27e92-4b79-419a-9bd7-800bc7773b6f_isr_1040013_11.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb0dc3328fa65aee70d12e58f3fdb6ba with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash ce5d1e2b2f25a768bf65f6cd61d810e8 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5f9c7a18-af6b-4f2b-b39c-00eea0169aae_characterizeImage_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5f9c7a18-af6b-4f2b-b39c-00eea0169aae_characterizeImage_1040013_8.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3c488229-5cdf-4f3f-9e61-e8e6f3c495f8_isr_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3c488229-5cdf-4f3f-9e61-e8e6f3c495f8_isr_1040013_8.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8d907d3d6a02df4c6b6b70b60fb8697 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e2ab276980124150c7632c94e74432be parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/61d52d01-d538-4492-884a-35c3efa7e4df_characterizeImage_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/61d52d01-d538-4492-884a-35c3efa7e4df_characterizeImage_1039909_43.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/59308bae-1143-4a41-a7f6-e284223d65ee_isr_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/59308bae-1143-4a41-a7f6-e284223d65ee_isr_1039909_43.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a029ee1209692a8280a6bb38b900bf1a with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash e09bd6826be112625c709cd399383d2e parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c636f9b6-d6e9-4870-a6b2-379a053f85fb_characterizeImage_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c636f9b6-d6e9-4870-a6b2-379a053f85fb_characterizeImage_1039908_14.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/aa135a62-2129-41c7-bc61-77b43ab921c2_isr_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/aa135a62-2129-41c7-bc61-77b43ab921c2_isr_1039908_14.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 363db0d5041c376ef61b3181308d33b4 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 0a901406c7351950e593efe4088de667 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/062d865d-bec5-4d63-861f-17ca42e8ea88_characterizeImage_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/062d865d-bec5-4d63-861f-17ca42e8ea88_characterizeImage_1039907_41.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/161ec3c7-5485-4cad-ab95-424e6a402ac1_isr_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/161ec3c7-5485-4cad-ab95-424e6a402ac1_isr_1039907_41.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b03dd02e8efcdc5c6cf11883980f00 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash d4cba0e8143075db3527398f1e1c595f parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/139b6178-b9a1-403c-97e3-7585ea1b6e28_characterizeImage_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/139b6178-b9a1-403c-97e3-7585ea1b6e28_characterizeImage_1039907_33.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d6365ac7-ddd8-433f-a670-d5b748d25de1_isr_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d6365ac7-ddd8-433f-a670-d5b748d25de1_isr_1039907_33.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dcfddc7c1ae0fe7985ad1cc8e827d76 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 02e3601104613f6f7ec7c20b7e5b7a26 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/62b90bfe-e53a-45da-be53-95c63f153941_characterizeImage_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/62b90bfe-e53a-45da-be53-95c63f153941_characterizeImage_1039908_5.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/83a2d955-7b18-4aca-a1b9-d77e3ddf3a06_isr_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/83a2d955-7b18-4aca-a1b9-d77e3ddf3a06_isr_1039908_5.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f79a23370568aa08f3cf06fe3a2b1c43 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash f2f47368dece0447bb5e32b91f95f79c parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/27eb8fac-7e4f-4ddc-94ef-6f69127531e7_characterizeImage_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/27eb8fac-7e4f-4ddc-94ef-6f69127531e7_characterizeImage_1039907_8.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2be880e-3576-4247-9bb2-f1f8ced31dd1_isr_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2be880e-3576-4247-9bb2-f1f8ced31dd1_isr_1039907_8.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 821ae211feec246302b06bf56918caab with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash d101393efa07eabfdd42429c686c1b07 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/75f89dc5-71e7-4a49-b06d-bd0c90239c36_characterizeImage_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/75f89dc5-71e7-4a49-b06d-bd0c90239c36_characterizeImage_1039909_56.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d20046ce-be89-432b-88fd-2c5da7884e73_isr_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/d20046ce-be89-432b-88fd-2c5da7884e73_isr_1039909_56.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f01fb4327421307263e2ea4a97ea61 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash b7c3d4280a3e2c55aaab439b3dbb3bc6 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a4377e4b-349e-413e-a925-914c6302b2fe_characterizeImage_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a4377e4b-349e-413e-a925-914c6302b2fe_characterizeImage_1039908_41.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/758125a7-7ce4-41a5-926a-60e8cab0e2e9_isr_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/758125a7-7ce4-41a5-926a-60e8cab0e2e9_isr_1039908_41.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b675c7dbc7616b61c96f13d054723c03 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ffc8f03c9203fb36e251e3fa487945ec parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d92d25d3-04f3-499c-8a38-751d141c1cc7_characterizeImage_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d92d25d3-04f3-499c-8a38-751d141c1cc7_characterizeImage_1039908_27.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4ad97c2c-edb1-4d78-bf3c-24333a3ffcb3_isr_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4ad97c2c-edb1-4d78-bf3c-24333a3ffcb3_isr_1039908_27.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d07e2f00963421575be01175d843ea3 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 2bb08bc177d7df2334b3d3f523c7b72c parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/67c0b68a-b39d-4de9-b695-485cd688da08_characterizeImage_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/67c0b68a-b39d-4de9-b695-485cd688da08_characterizeImage_1039909_27.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1f5dbdeb-532d-400b-a47e-b12c295da03d_isr_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1f5dbdeb-532d-400b-a47e-b12c295da03d_isr_1039909_27.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff8aa063ab924f5efd8c590aa5b700a with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 04d38860daedadf4544f924ac3068539 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cffe65fd-be45-4b5c-ab75-58bc7f436dc7_characterizeImage_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cffe65fd-be45-4b5c-ab75-58bc7f436dc7_characterizeImage_1039908_47.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b2984339-8fa0-4100-b874-c78782bcf073_isr_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b2984339-8fa0-4100-b874-c78782bcf073_isr_1039908_47.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61196d19c2ec9fd977278b8f164fc3f5 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 752b8c5a0d59ef3b02c3ff428a95e3fc parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b40e686-6722-49a2-8d23-7178afed9c56_characterizeImage_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b40e686-6722-49a2-8d23-7178afed9c56_characterizeImage_1039909_40.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3a6f466e-fd2d-4d95-bdba-83dc25407bb9_isr_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3a6f466e-fd2d-4d95-bdba-83dc25407bb9_isr_1039909_40.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab550904b2c7e2d5c53d6a6c2a8491a2 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 1d21b4f9d475175587a1c90601477bb0 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f3c3b1bd-e5cb-4139-97ee-5cc18cb79ad3_characterizeImage_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f3c3b1bd-e5cb-4139-97ee-5cc18cb79ad3_characterizeImage_1039909_39.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/56abe176-d7de-48a6-b59d-3aadf4c65351_isr_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/56abe176-d7de-48a6-b59d-3aadf4c65351_isr_1039909_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:16 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7cfff8553778a5b6c339efefa4cf66 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 3268ff95746ce7831c37a55dc6cc9c64 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/af308e70-253e-40e3-b37b-70d1eabccc86_characterizeImage_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/af308e70-253e-40e3-b37b-70d1eabccc86_characterizeImage_1039908_36.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/27ddb09d-a91d-4568-ace4-aa56520becfe_isr_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/27ddb09d-a91d-4568-ace4-aa56520becfe_isr_1039908_36.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39bc5d3de70b5233538fa458aedca93c with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash d805df08a8d927396eab9928c7a6b58f parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/21f739ea-27ce-47ec-82b6-9a8ba94b8b51_characterizeImage_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/21f739ea-27ce-47ec-82b6-9a8ba94b8b51_characterizeImage_1039908_33.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26ade41f-969b-44bc-a6b3-cd4e8d5e52c8_isr_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26ade41f-969b-44bc-a6b3-cd4e8d5e52c8_isr_1039908_33.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359a7ef0bbbb045806e8a621aa04aa5b with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 779dad62d8e9f086ab81de9f01542443 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3115e165-a899-436d-8705-f2c3847f8a44_characterizeImage_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3115e165-a899-436d-8705-f2c3847f8a44_characterizeImage_1040013_41.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6cb98973-8699-409c-a2ee-dfa446e99bf5_isr_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6cb98973-8699-409c-a2ee-dfa446e99bf5_isr_1040013_41.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea34e850d1f66d03ede4d1c9c3c103a9 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash bb4eafb93fe1d4849e13433c31fd54e4 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/d297764d-8e64-4a87-9c5a-7fc03f86ede4_characterizeImage_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/d297764d-8e64-4a87-9c5a-7fc03f86ede4_characterizeImage_1039907_12.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f79611db-466a-4418-9838-6c8a639d6f01_isr_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f79611db-466a-4418-9838-6c8a639d6f01_isr_1039907_12.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc5ac018b0d80303a3302bec352776ed with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 5ebe4a314eccb8178a8c01940f544dac parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5a2aff30-b933-4e36-be1a-0f6345a24448_characterizeImage_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5a2aff30-b933-4e36-be1a-0f6345a24448_characterizeImage_1039909_29.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b1569d44-daf6-49f2-aca2-7009716a7074_isr_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b1569d44-daf6-49f2-aca2-7009716a7074_isr_1039909_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:17 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7410b7dce61cfc405df5aaf22efff657 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 8f5ac92d14f513bb520bde52d0f3f737 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/54c101cf-02d5-473f-88b2-38b5e572e9f6_characterizeImage_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/54c101cf-02d5-473f-88b2-38b5e572e9f6_characterizeImage_1039908_56.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cb2a59ad-66ed-4347-8584-9210804c0641_isr_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cb2a59ad-66ed-4347-8584-9210804c0641_isr_1039908_56.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb0a795e559537ef49d33df143af9ef2 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 024708ff8a587b87e96582fafd8356f7 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0af03258-01cb-42a1-9c2d-59613b354cd5_characterizeImage_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0af03258-01cb-42a1-9c2d-59613b354cd5_characterizeImage_1039908_17.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/03536243-73a2-4366-bbc9-41d382444d9f_isr_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/03536243-73a2-4366-bbc9-41d382444d9f_isr_1039908_17.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 502b9cedc2bf5fb29e9cc7064e273c4e with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 399ca78267da9ed7848f6d1b523ff6fa parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81698a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a89830e6-b08d-4499-bf5d-90d929d588ea_characterizeImage_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a89830e6-b08d-4499-bf5d-90d929d588ea_characterizeImage_1040013_46.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e760bb92-0f58-481c-81e1-7424c7f26fa3_isr_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e760bb92-0f58-481c-81e1-7424c7f26fa3_isr_1040013_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:17 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:18 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112ec71a1b3830686528cbc7afd6cec9 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash e261e300468228411cd2b5b2914327d4 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1d0ace9b-7da4-4859-9ecf-7b686a79c433_characterizeImage_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1d0ace9b-7da4-4859-9ecf-7b686a79c433_characterizeImage_1039909_37.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9d80951b-6e8e-4207-8fa6-e343efa757ea_isr_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9d80951b-6e8e-4207-8fa6-e343efa757ea_isr_1039909_37.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7b70dfd5c347cb70bac63f2b59c0ff1 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 80955f6c0dcf67f8100aae1c122b7a95 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f12a8d9-cff8-450e-b022-a87ff6bc301f_characterizeImage_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f12a8d9-cff8-450e-b022-a87ff6bc301f_characterizeImage_1039909_54.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/54e8ac67-0214-48bc-bf02-825eca57ba9a_isr_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/54e8ac67-0214-48bc-bf02-825eca57ba9a_isr_1039909_54.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d0b4dba475507fc54d6fcfa6e94c89 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash b7f2ea0f12813485ac47c5892fdfd55a parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8fca7a7a-bfd3-4335-9246-402d0ac66d74_characterizeImage_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8fca7a7a-bfd3-4335-9246-402d0ac66d74_characterizeImage_1039909_9.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/152ee5dd-2482-4188-bd93-cbed7f5d4c92_isr_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/152ee5dd-2482-4188-bd93-cbed7f5d4c92_isr_1039909_9.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2779ecd884c1711ba25743ba9d355a9 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 73b7a8ec388539a8525a3e54233f2fcc parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a349c54d-0ede-4678-b935-5f0a402c8082_characterizeImage_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a349c54d-0ede-4678-b935-5f0a402c8082_characterizeImage_1039909_16.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/6d50b679-ac19-429a-a6f4-4871b95393db_isr_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/6d50b679-ac19-429a-a6f4-4871b95393db_isr_1039909_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:18 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1872520bd04684e68f8c512988233d25 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash c4fe3b38229afe37162eafece727fad2 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5879ce49-6958-4d35-bfc3-5abcbf5d0744_characterizeImage_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5879ce49-6958-4d35-bfc3-5abcbf5d0744_characterizeImage_1040013_58.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/37970f03-522e-416b-9b5d-4c6d478a9584_isr_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/37970f03-522e-416b-9b5d-4c6d478a9584_isr_1040013_58.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b95687bdbedfc7003785375059ff36f9 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 823f8539211999af5e7372f416a2c34f parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7c5b7240-f2f9-4ecd-923a-4f8a32711fad_characterizeImage_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7c5b7240-f2f9-4ecd-923a-4f8a32711fad_characterizeImage_1039908_35.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4f920359-ce1d-4351-af5e-ea178539f654_isr_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/4f920359-ce1d-4351-af5e-ea178539f654_isr_1039908_35.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bf85b326c5111ea780b3a826582aaeb with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 40c0e5125010df529d6b65b2c1352b5a parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/2cad68d3-d8eb-4a5e-b90c-c82f4e6b9c67_characterizeImage_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/2cad68d3-d8eb-4a5e-b90c-c82f4e6b9c67_characterizeImage_1039907_1.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/0eb58d83-5bc7-45e7-ace7-9976fd1b1181_isr_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/0eb58d83-5bc7-45e7-ace7-9976fd1b1181_isr_1039907_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:19 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:19 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:20 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:20 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:20 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:21 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:21 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:21 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:22 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:22 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b6677be805c2df9979b43ec505815b with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 25ccea73d8b4daeb2cdc60e5c895d3bc parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f0ac72f0-02b3-4ea6-b0e0-f20e40c36f94_characterizeImage_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f0ac72f0-02b3-4ea6-b0e0-f20e40c36f94_characterizeImage_1039907_52.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2bf69f57-252f-4a5d-b769-2e955b3026f6_isr_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/2bf69f57-252f-4a5d-b769-2e955b3026f6_isr_1039907_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:22 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:22 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ddd56958b79dcba6e44f712ccfd8cc8 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 5b48c15d429cf86b1823387cb2827d7e parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4438386a-b2a7-45fa-a9d8-02ea5d670e61_characterizeImage_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4438386a-b2a7-45fa-a9d8-02ea5d670e61_characterizeImage_1040013_1.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/0c0b16d3-e541-4d30-a7cc-b9425971c879_isr_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/0c0b16d3-e541-4d30-a7cc-b9425971c879_isr_1040013_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:23 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd901185394e95c2de56089be853fdbd with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 29e5a97577e651f0ee8f5ea1f6528919 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6a746962-b052-4054-884a-e9e3fcdd9a4e_characterizeImage_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6a746962-b052-4054-884a-e9e3fcdd9a4e_characterizeImage_1039909_30.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9a6af637-c193-4b01-9e0e-2e814ebc799a_isr_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9a6af637-c193-4b01-9e0e-2e814ebc799a_isr_1039909_30.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1472177b2c7fe6555046959f45ee03be with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 9f153c14f51fd4ceeb3cb5a9f329a1a8 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/38fe02b5-0c6b-4e68-8d32-73260afe162c_characterizeImage_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/38fe02b5-0c6b-4e68-8d32-73260afe162c_characterizeImage_1040013_25.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/448414ee-e998-4961-987b-76239b24e69f_isr_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/448414ee-e998-4961-987b-76239b24e69f_isr_1040013_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:23 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:23 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c38ebed3df80fb462fadad2546aaa8d with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 342858d2be62085ffb45a8427efa2341 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce3507b4-3d25-472d-9e16-f5004399d026_isr_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce3507b4-3d25-472d-9e16-f5004399d026_isr_1039907_61.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1d2ac776194f6897d6fb3442ce5905 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 28ab71a4e222f92ac8fb949aee29f933 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/940149e9-198b-4edd-9bc3-e7869286fb2a_isr_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/940149e9-198b-4edd-9bc3-e7869286fb2a_isr_1040013_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:24 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc325acc509b005f3866f06769a1268a with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 236b427db1334751b6b11ebdbd983b6e parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3945953-7241-42e1-9131-1c1eeee5f936_characterizeImage_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3945953-7241-42e1-9131-1c1eeee5f936_characterizeImage_1039909_33.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/60e1cd64-f1b8-4f68-ac8f-5ad8071116c1_isr_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/60e1cd64-f1b8-4f68-ac8f-5ad8071116c1_isr_1039909_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:24 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4bc320a2b86dba8862bf8a6dfcd1c6f with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 2f651a0a20532ee75d81a74513942daf parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ff276af2-b134-4f52-89eb-ef8a10b757cd_characterizeImage_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ff276af2-b134-4f52-89eb-ef8a10b757cd_characterizeImage_1040013_6.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f2ccf9c5-658b-412a-b6c3-92db5ac0908a_isr_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f2ccf9c5-658b-412a-b6c3-92db5ac0908a_isr_1040013_6.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 172b5a9d3b9827287fdd3b312014686b with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 49b41c5a724c006822054417ae7f3eb1 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/49a22813-172d-4c1d-8f9c-17493c4dd0fa_characterizeImage_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/49a22813-172d-4c1d-8f9c-17493c4dd0fa_characterizeImage_1039908_12.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9a84e977-d3e5-4c0d-a17e-e643e9f99117_isr_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9a84e977-d3e5-4c0d-a17e-e643e9f99117_isr_1039908_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:25 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ad7c53130041f1e9a164486ece46d46 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash fb3b3e3726d65abc9edb6099461e7e2a parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/cebe6ec8-a21d-4f47-ad38-e3f031e63b20_characterizeImage_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/cebe6ec8-a21d-4f47-ad38-e3f031e63b20_characterizeImage_1039907_54.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/64b42ac4-8060-4897-a937-7326d7b17ce5_isr_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/64b42ac4-8060-4897-a937-7326d7b17ce5_isr_1039907_54.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c01904ec14f31c9404e00a9f8f877d7c with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 209fd5f465b141f90c711083d36b08b0 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dfb32041-7c35-4c18-b34f-cf2cceceb59b_characterizeImage_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dfb32041-7c35-4c18-b34f-cf2cceceb59b_characterizeImage_1040013_57.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6624c00a-d10b-4531-9919-c7f29274c884_isr_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6624c00a-d10b-4531-9919-c7f29274c884_isr_1040013_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:25 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9e3cb503fc8ab0666d3ce6bf85f9a26 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 169e28f1318d1b2a4dac9265a533b290 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b4925220-5f06-4c34-8103-051b0f5c8e9e_characterizeImage_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b4925220-5f06-4c34-8103-051b0f5c8e9e_characterizeImage_1040013_51.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7795da99-66c7-4222-9813-a8b5790e88da_isr_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7795da99-66c7-4222-9813-a8b5790e88da_isr_1040013_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:26 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6666f7c1a135c1f349c5c4cad4be81 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash ebafc9785029a3996b2e3352b4170aa0 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/757a8220-1895-44c6-ba95-af178355b019_characterizeImage_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/757a8220-1895-44c6-ba95-af178355b019_characterizeImage_1039909_7.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/63a8ba8b-bfa8-4f4d-a83f-c40bb063ff8c_isr_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/63a8ba8b-bfa8-4f4d-a83f-c40bb063ff8c_isr_1039909_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:26 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:26 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ff3ace283180df8bd0b772755f8776 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 2bfb14fdf253c42471dbb5e004e012ec parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/bc313123-d9a0-4150-a1da-95a7c4a0e60a_characterizeImage_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/bc313123-d9a0-4150-a1da-95a7c4a0e60a_characterizeImage_1039908_11.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/36f19dd3-66fc-4415-9c09-06e649280954_isr_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/36f19dd3-66fc-4415-9c09-06e649280954_isr_1039908_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:26 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe02e222b2cc78af286e7c4b44c586c4 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 85c62d9e2d8f0ba2d43783177e2138e0 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4fe56190-09fe-43cf-b474-68644b2a49af_characterizeImage_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4fe56190-09fe-43cf-b474-68644b2a49af_characterizeImage_1040013_56.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e0e44427-ddc2-42e0-9973-cf4f6dcf5300_isr_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e0e44427-ddc2-42e0-9973-cf4f6dcf5300_isr_1040013_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b34336e73510aa57787d88436bf2e1a with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 6792e621c9e86a5ee0d0a43cdca5a616 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8e85db91-e8c8-4d48-abff-ffdcd816f09b_characterizeImage_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8e85db91-e8c8-4d48-abff-ffdcd816f09b_characterizeImage_1039909_24.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/057a6bac-c21a-4fbe-83b8-40d2f7826c37_isr_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/057a6bac-c21a-4fbe-83b8-40d2f7826c37_isr_1039909_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ac8105b007715748412ad0f9447dda with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 39bc61e17dd31c49779dee3a3e2d522c parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/990258c9-e84c-45c9-8186-fcb2f3c99996_characterizeImage_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/990258c9-e84c-45c9-8186-fcb2f3c99996_characterizeImage_1040013_4.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/195d7175-6eda-4478-983f-e8b3845b7623_isr_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/195d7175-6eda-4478-983f-e8b3845b7623_isr_1040013_4.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b400aab721897a13495a95f8be3a846 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 956652e87e0b18d95d843e70f9ee7159 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/3b43458c-457b-46b5-8d51-c61947538c3c_characterizeImage_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/3b43458c-457b-46b5-8d51-c61947538c3c_characterizeImage_1039909_19.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/ed391938-d2ba-457d-a60d-42e56ba3fe4d_isr_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/ed391938-d2ba-457d-a60d-42e56ba3fe4d_isr_1039909_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39b3a646005a759522dc0393329c615 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash bfa76dcfc155b505cfbc23148c972dee parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/8a8a6287-4ecb-40b0-978d-52f787a92285_characterizeImage_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/8a8a6287-4ecb-40b0-978d-52f787a92285_characterizeImage_1039908_25.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/184f7737-b873-465e-beb7-48cbff9d7f58_isr_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/184f7737-b873-465e-beb7-48cbff9d7f58_isr_1039908_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:27 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 720ca1041361a056740d3760a99d67ba with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash de784107e4782bf05eaf7517c7278a24 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/81feed20-88f8-4401-8f0b-4e6884fdc37f_characterizeImage_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/81feed20-88f8-4401-8f0b-4e6884fdc37f_characterizeImage_1039909_14.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2bd54027-7300-4cb4-895f-c8946e09fc97_isr_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2bd54027-7300-4cb4-895f-c8946e09fc97_isr_1039909_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:28 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 167d2284e6abaf57977ae16c510d4fc5 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 14ef74dd8460385a858270d1f1b37de9 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aa2d8a32-4d7a-4bf3-ad2b-11f8ee3b046c_characterizeImage_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aa2d8a32-4d7a-4bf3-ad2b-11f8ee3b046c_characterizeImage_1040013_38.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1f97f9a8-bbc3-4b8d-98aa-f2789b226dd2_isr_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1f97f9a8-bbc3-4b8d-98aa-f2789b226dd2_isr_1040013_38.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 404ac1b7bc6507c9c87659c16f052712 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 84aa13b4c24a7dfc511e706686f705fa parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1754517e-482b-4a0e-8809-abeb9158392e_characterizeImage_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1754517e-482b-4a0e-8809-abeb9158392e_characterizeImage_1040013_54.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d619eddc-f5a0-4515-9a65-c20d4b05e941_isr_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d619eddc-f5a0-4515-9a65-c20d4b05e941_isr_1040013_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:28 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9082cba1d1dbc119b19a1fd9fd79602c with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 285ecb9337ee3ac60b83083f301595a4 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a2ed190f-bb30-4edd-a36b-4abd0cd80069_characterizeImage_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a2ed190f-bb30-4edd-a36b-4abd0cd80069_characterizeImage_1039909_62.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/49c3563f-b648-48b8-b51c-a03d3ca8545d_isr_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/49c3563f-b648-48b8-b51c-a03d3ca8545d_isr_1039909_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:29 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96853752df4f14add27a0f7d4f4fdb46 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 3014242bd585bc34936a7aabc3360f60 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d85c9485-a913-4d12-88fa-5f9b25ab4eb2_characterizeImage_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d85c9485-a913-4d12-88fa-5f9b25ab4eb2_characterizeImage_1039909_47.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/faba9a61-be97-4b84-80d9-36f4d4301e04_isr_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/faba9a61-be97-4b84-80d9-36f4d4301e04_isr_1039909_47.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c713f851ff3afa1b408d46682a69dc00 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 94d16236045dbf3b1bfd3af464dead73 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/58cc4737-6ae8-4028-a540-cdfb40d51074_characterizeImage_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/58cc4737-6ae8-4028-a540-cdfb40d51074_characterizeImage_1039907_34.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/70c4520c-9a75-4108-a263-accad52c71e8_isr_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/70c4520c-9a75-4108-a263-accad52c71e8_isr_1039907_34.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3626b63f0aa6ec3f1ee86459e60a8731 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 0ca5729918013c6c6251f94dc60ba8c8 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26636b0c-0928-4614-b996-8a4fe773e99c_isr_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/26636b0c-0928-4614-b996-8a4fe773e99c_isr_1039908_61.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c63ba7cf7446a929db116a0c2b142bd8 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 5481614ac04765c629f5fc8954acebe3 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c5831da2-b309-4372-97a8-025866ec0406_characterizeImage_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c5831da2-b309-4372-97a8-025866ec0406_characterizeImage_1039909_21.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8f77ce2a-82b5-4897-a94c-69463245ce63_isr_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8f77ce2a-82b5-4897-a94c-69463245ce63_isr_1039909_21.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68565531cdd18ec8c851b6d2a3b3a60d with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 648d6d1ecfde161e2cb99959c591187a parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/85c9f9b6-c657-4339-8499-9f31ce20e544_characterizeImage_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/85c9f9b6-c657-4339-8499-9f31ce20e544_characterizeImage_1040013_16.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/08c6e99e-9223-403a-906e-dfc7d7e60141_isr_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/08c6e99e-9223-403a-906e-dfc7d7e60141_isr_1040013_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:30 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdb56036b0475f9db9d0bdbec4cdc2de with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 506f787fb195864fafb84a5200fea608 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/e3d11a5b-c083-4789-8cf8-5b7bbe82cd82_characterizeImage_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/e3d11a5b-c083-4789-8cf8-5b7bbe82cd82_characterizeImage_1039907_28.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/911930fd-b87f-4380-9fd6-b2cbc2aa46f4_isr_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/911930fd-b87f-4380-9fd6-b2cbc2aa46f4_isr_1039907_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:30 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44350b7b255f20519d2963d7e05460a0 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash f4a76527ad9f5f982207a2d7d60ef11c parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/81663b88-188c-43f8-9286-61e59bff6a57_characterizeImage_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/81663b88-188c-43f8-9286-61e59bff6a57_characterizeImage_1039908_55.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/5f41386f-13aa-44ca-aedc-d9e8e7f6c409_isr_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/5f41386f-13aa-44ca-aedc-d9e8e7f6c409_isr_1039908_55.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16002e13f3eeeaed3eebba4a6d2fc39e with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 8aa3b55ae4f8602ed041e5e3ca349b71 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f15f4332-64a8-4d02-82ac-03f98ef484c6_characterizeImage_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f15f4332-64a8-4d02-82ac-03f98ef484c6_characterizeImage_1040013_24.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26694093-ca9d-47d3-a149-251c89d4ecd1_isr_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26694093-ca9d-47d3-a149-251c89d4ecd1_isr_1040013_24.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a680b4b0bbeb24c9449d5740a8d1ab3a with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 8e084e3eb97e2308f4ce1d76fc7d2a47 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/11178b2d-0e10-447d-b40b-b042a37dc27f_characterizeImage_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/11178b2d-0e10-447d-b40b-b042a37dc27f_characterizeImage_1040013_34.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/81d0fafd-1932-4c4b-bf41-624e817998f1_isr_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/81d0fafd-1932-4c4b-bf41-624e817998f1_isr_1040013_34.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3e85eb7b29392d88a85597126559e6a with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 2d8f7b13f213b58e673ed87fd3b3a4c8 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fc3a5e9a-2f1c-4667-8059-10c0a260e29b_characterizeImage_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fc3a5e9a-2f1c-4667-8059-10c0a260e29b_characterizeImage_1039908_3.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/05f74dc0-46f3-443f-b26c-f8650537a12d_isr_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/05f74dc0-46f3-443f-b26c-f8650537a12d_isr_1039908_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195151 PD 17195152 PD 17195153 PD 17195154 PD 17195155 PD 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 57/5 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 57/5 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f224fb639a23ea2bd2a74ec746ee957 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 04ba8a72d2c867dc26cbe0889abf559e parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c0f50cf3-9d01-4c53-a480-723a0a7e0f11_characterizeImage_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c0f50cf3-9d01-4c53-a480-723a0a7e0f11_characterizeImage_1039909_52.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/79ebfb58-3556-42ff-baad-47cf36ac8329_isr_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/79ebfb58-3556-42ff-baad-47cf36ac8329_isr_1039909_52.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffa642ad6f99da24989d9c80c9d25d61 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 3306e4892f2600b3d597fdef97677164 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6badb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c9421425-c7d3-4c2d-af8d-5891db14c9f6_characterizeImage_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c9421425-c7d3-4c2d-af8d-5891db14c9f6_characterizeImage_1039908_31.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/824cafdf-7d52-4d15-9e47-8091f21401e1_isr_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/824cafdf-7d52-4d15-9e47-8091f21401e1_isr_1039908_31.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c87066e1b5c05c430bfefc05b2fbd88 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 272a6cf8fee9fc7c4db4525a7071ba06 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/328ade1c-5e0f-4de1-ada3-25946310e3b2_characterizeImage_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/328ade1c-5e0f-4de1-ada3-25946310e3b2_characterizeImage_1039907_59.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/84045db3-1751-451a-91af-f63d0bf0aa83_isr_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/84045db3-1751-451a-91af-f63d0bf0aa83_isr_1039907_59.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a0327caba67197cf9e6324462bc0e3b with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 1b64622d3aade22ae7b24f04f0943a16 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ee127dc9-41b9-4df7-bad7-d07e275d6ffd_characterizeImage_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ee127dc9-41b9-4df7-bad7-d07e275d6ffd_characterizeImage_1039907_48.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/387f3847-1ec0-4ef7-9804-23fbb2780bbd_isr_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/387f3847-1ec0-4ef7-9804-23fbb2780bbd_isr_1039907_48.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4437cc43d93b0404b2b20fdb734365 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 6e74a81e918bef8de7f41d213fdfc867 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0a392c9c-762b-4a4b-b72a-aeba432f1154_characterizeImage_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0a392c9c-762b-4a4b-b72a-aeba432f1154_characterizeImage_1039908_38.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cd6dbfce-bbb9-4cf7-8398-601c3566c38c_isr_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cd6dbfce-bbb9-4cf7-8398-601c3566c38c_isr_1039908_38.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44e83b3076218ffc1e8be6ad79e05177 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash e7cf0afe329d39487790d3f95aeca5ed parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7890fba9-872a-4ef5-8205-80f08ad4dc02_characterizeImage_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7890fba9-872a-4ef5-8205-80f08ad4dc02_characterizeImage_1039909_1.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1065e7c4-ba68-4237-b2e6-c6674530915e_isr_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1065e7c4-ba68-4237-b2e6-c6674530915e_isr_1039909_1.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc813be1301e1b17579c4a7fc3f691e with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash cc6c798f4b701cda0e03e4b961ed5949 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a277e9fa-d6ff-4466-abd8-159712c51eec_characterizeImage_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a277e9fa-d6ff-4466-abd8-159712c51eec_characterizeImage_1039908_40.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9f82443f-9923-4d62-a2a9-deeec9e7f7cb_isr_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9f82443f-9923-4d62-a2a9-deeec9e7f7cb_isr_1039908_40.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd3ebfe2abc1466142d211f90a1d47e5 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 5152b6b362bd3cc08499d3b6d6945da2 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a44ec457-a316-45e7-9586-2ed091f59147_characterizeImage_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a44ec457-a316-45e7-9586-2ed091f59147_characterizeImage_1040013_50.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2c184267-e1e3-4355-9c3d-544e0680cff3_isr_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2c184267-e1e3-4355-9c3d-544e0680cff3_isr_1040013_50.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b321d1c9ec4f56d7679aa6c8e7ad00b with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash adb721cfa3aa30e2b840466b9804f357 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/06fc3381-a027-4a62-9226-65e5c8ddb1c3_characterizeImage_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/06fc3381-a027-4a62-9226-65e5c8ddb1c3_characterizeImage_1040013_44.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/771bc922-f47c-47c8-8edd-aeae737e7caa_isr_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/771bc922-f47c-47c8-8edd-aeae737e7caa_isr_1040013_44.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce83d692bf9d456c6293a4c02374d96e with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 5aaa5d790a05b8d91b25bbba778c2651 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0c69e746-52c2-441d-9106-9d7f107611a3_characterizeImage_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0c69e746-52c2-441d-9106-9d7f107611a3_characterizeImage_1039907_35.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/825c2df7-d680-4263-a368-a8d40cbd373c_isr_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/825c2df7-d680-4263-a368-a8d40cbd373c_isr_1039907_35.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d5ddae67ba9323207456f123d841472 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash b534662597802c61c8a054d5a13569b8 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/669ee119-7448-490b-8ba9-0b14783eb255_characterizeImage_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/669ee119-7448-490b-8ba9-0b14783eb255_characterizeImage_1039908_13.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b038ba2e-6a31-4447-bcc0-d400c79c7866_isr_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b038ba2e-6a31-4447-bcc0-d400c79c7866_isr_1039908_13.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4654c9f3f3d59d00283968aa4c343d with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 15b264d09e9d165d4762ec1dfb954437 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/48e2e3ab-d285-4f01-9192-56de12169287_characterizeImage_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/48e2e3ab-d285-4f01-9192-56de12169287_characterizeImage_1040013_18.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ab100783-f33f-4488-aac7-9d03a5b50e59_isr_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ab100783-f33f-4488-aac7-9d03a5b50e59_isr_1040013_18.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eab06dac5e554558abc215459e093ea with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 7810b5be86b860e689355ae783e7a59a parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/20068437-9529-4885-9350-c2f2e2a2d6ad_characterizeImage_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/20068437-9529-4885-9350-c2f2e2a2d6ad_characterizeImage_1039908_9.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/2cc06d3a-b565-4378-b0a4-9ff76adaff43_isr_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/2cc06d3a-b565-4378-b0a4-9ff76adaff43_isr_1039908_9.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9160b292b27a72b51f026599301489ef with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash b17f8efe4721a51dc949bffdaac5d5da parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7b8dd8b4-7bc0-4174-ba8b-bb726c5c6464_characterizeImage_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7b8dd8b4-7bc0-4174-ba8b-bb726c5c6464_characterizeImage_1039907_3.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4db7e558-cc04-4526-b9e1-7408ab51b948_isr_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4db7e558-cc04-4526-b9e1-7408ab51b948_isr_1039907_3.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ece896f6d209840de46a6d24779ddb11 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 039f15cafbc0cd53f5d05b587acd6f8e parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f3dea3a0-d55f-41ac-820d-7b6f969a104d_characterizeImage_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f3dea3a0-d55f-41ac-820d-7b6f969a104d_characterizeImage_1039907_57.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d38d8cf4-efd6-41fd-9425-3ed14fe55327_isr_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d38d8cf4-efd6-41fd-9425-3ed14fe55327_isr_1039907_57.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3c1f44c8687ec3b425eec32793201a9 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 80a72e7d30eb1c0fd946e7893ea287b5 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/92000987-b047-403b-a2a9-7efd76598748_characterizeImage_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/92000987-b047-403b-a2a9-7efd76598748_characterizeImage_1039907_60.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5a55299b-edbc-41b7-8743-95b1914604bf_isr_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5a55299b-edbc-41b7-8743-95b1914604bf_isr_1039907_60.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 614580f3128c021f29a190311dabaa3e with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash f7e82dc35339981b61c2d550757bf4da parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bade40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2de57b8b-39c1-45e2-9d11-e3cff2c65bce_characterizeImage_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2de57b8b-39c1-45e2-9d11-e3cff2c65bce_characterizeImage_1039909_26.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/edbc39e9-2465-4299-9b39-6dfdb3bad5b7_isr_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/edbc39e9-2465-4299-9b39-6dfdb3bad5b7_isr_1039909_26.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d6e9b913ff679d5b350492ced088943 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash c0cc338c87da79fc80aca6a00231d731 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5165940e-8e8c-408b-92cf-51ee0c0f34d9_characterizeImage_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5165940e-8e8c-408b-92cf-51ee0c0f34d9_characterizeImage_1039907_29.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f52f6f0d-c4e8-4fa4-8774-d5292b467603_isr_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f52f6f0d-c4e8-4fa4-8774-d5292b467603_isr_1039907_29.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 999550807bc2c6ee8197d2f5562b9600 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 685398addbe8fdb31fab313077a04d4a parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/b163ed18-b6d7-46cf-ba1f-dd744b0ce132_characterizeImage_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/b163ed18-b6d7-46cf-ba1f-dd744b0ce132_characterizeImage_1039908_58.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6779707d-df87-4880-9e9b-6f2a77795ed2_isr_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6779707d-df87-4880-9e9b-6f2a77795ed2_isr_1039908_58.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa0f04c909470a69d79936e506714aa with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 8efce52be14163ad9eba64d702ba3134 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1dd176e2-0a79-42df-8753-c95ee86d1061_characterizeImage_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1dd176e2-0a79-42df-8753-c95ee86d1061_characterizeImage_1039907_10.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4d4d7f10-074f-44b8-a1b4-18b4b9344546_isr_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/4d4d7f10-074f-44b8-a1b4-18b4b9344546_isr_1039907_10.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c951172c851e96ea0af450f4e240cd with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 687538a8cbdd1bba71fc18cf8ee8e4a6 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/54ae6c08-af1e-44e8-9377-ccd4bbeea691_characterizeImage_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/54ae6c08-af1e-44e8-9377-ccd4bbeea691_characterizeImage_1040013_60.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e4527795-6b09-46b5-8d39-04f8ccbc14e2_isr_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/e4527795-6b09-46b5-8d39-04f8ccbc14e2_isr_1040013_60.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e567165538bd5da60c6bffff232737 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 26c3434d8fc12dfc276d29f1446b5fee parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/58f9b8c2-280d-4d20-bf2c-f95823230ab2_characterizeImage_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/58f9b8c2-280d-4d20-bf2c-f95823230ab2_characterizeImage_1039909_58.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2d915c97-f801-4c68-892c-94b816bb47a7_isr_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2d915c97-f801-4c68-892c-94b816bb47a7_isr_1039909_58.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d009dd13612f4d10bc532fb17f3b17 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 5605d3fa0bd2a1065c433aee790080be parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bacea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f28596d-678f-4fdc-81be-f4eeba760a2a_characterizeImage_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f28596d-678f-4fdc-81be-f4eeba760a2a_characterizeImage_1039909_12.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bcedeab2-95d9-4cfd-ad9b-caebc61b826e_isr_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bcedeab2-95d9-4cfd-ad9b-caebc61b826e_isr_1039909_12.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2130215b2077fb8d843d66f9f781bd2e with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash bf6fac9baed9034ccf52375c9e881b7f parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bacb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dafa59bf-1839-41d0-9e3f-35acde20db4d_characterizeImage_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dafa59bf-1839-41d0-9e3f-35acde20db4d_characterizeImage_1039908_29.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/037c55f6-eed3-4b01-be3e-0092820e1272_isr_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/037c55f6-eed3-4b01-be3e-0092820e1272_isr_1039908_29.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39b60e78176f43407a752c3323be2006 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash cc87dcef0be0080c7329819e7855e780 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/118c894a-2afb-4b4c-8d11-af7ec5a1192a_isr_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/118c894a-2afb-4b4c-8d11-af7ec5a1192a_isr_1039909_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 57/5 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 600f6c40b0e8ea4c77db4d051aabed00 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash ec16ecbb73ec525ac5db56f2836abf23 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7612296f-8653-4c3c-a4e5-9e1c8d636533_characterizeImage_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7612296f-8653-4c3c-a4e5-9e1c8d636533_characterizeImage_1039907_38.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/747d5ed9-6547-445d-aa8e-7cc3fe270089_isr_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/747d5ed9-6547-445d-aa8e-7cc3fe270089_isr_1039907_38.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f44bfb3165534ffd0c30a04ef683bf8 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash f2d2a7c27f7afc9a1e7a3183b0ea6257 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/bb61843b-d702-410e-b06d-bb87c25e7cbe_characterizeImage_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/bb61843b-d702-410e-b06d-bb87c25e7cbe_characterizeImage_1039909_15.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5eb1b285-0114-4cb3-adaa-d7d2ffc92de5_isr_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5eb1b285-0114-4cb3-adaa-d7d2ffc92de5_isr_1039909_15.stderr 2024-03-18 05:21:41 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:21:41 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:41 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:42 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c708571d5b59687a65b7a671dabd9c4 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 1c2ba69a45fa0442ce848276e1cc94fe parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/03d9f05a-1237-40e8-866e-c220a53a175d_characterizeImage_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/03d9f05a-1237-40e8-866e-c220a53a175d_characterizeImage_1039909_45.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5c8bbd7a-0085-4b93-a49f-fcd418bb56e3_isr_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5c8bbd7a-0085-4b93-a49f-fcd418bb56e3_isr_1039909_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:42 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf3ef4b7ed171beadf4ca909770383a with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 03d8ac9af6680731de874090147ac7ae parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/e6767b79-0430-42f0-b71e-bc2a9bb2ebdc_isr_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/e6767b79-0430-42f0-b71e-bc2a9bb2ebdc_isr_1039909_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:42 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af59a1e68278bb2beca159830f25297 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash c211b57cf2982329d3a549c9f6d6a0bb parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c082c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cf9e7f65-07ca-4c6d-a553-844e6f083cc0_characterizeImage_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cf9e7f65-07ca-4c6d-a553-844e6f083cc0_characterizeImage_1040013_22.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/faec5d9f-fc36-44bf-8cc4-7b1f59b145c0_isr_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/faec5d9f-fc36-44bf-8cc4-7b1f59b145c0_isr_1040013_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:43 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ecceaba3728477ec8feda0e7316327 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 1f31575f990e81bee3191b30813e93d5 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c02a4e53-2997-402d-b51a-eedfa312a17d_characterizeImage_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c02a4e53-2997-402d-b51a-eedfa312a17d_characterizeImage_1039909_46.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04ade2c7-f51b-468d-8f72-c25c842cc796_isr_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04ade2c7-f51b-468d-8f72-c25c842cc796_isr_1039909_46.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc77df4d417217acc9152050d7fd251 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash ad03519895df4f1dc3bf601cd2900cb8 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cbcf0f3e-18eb-48a8-b165-45431fe379c2_characterizeImage_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cbcf0f3e-18eb-48a8-b165-45431fe379c2_characterizeImage_1039908_49.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/09aeadc4-adef-4d54-8731-30f7b43778fd_isr_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/09aeadc4-adef-4d54-8731-30f7b43778fd_isr_1039908_49.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16a6e7478171e0c5e538d5282e8394a4 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash a8e591e1191163124b522ab240ed1905 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c085e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/12088e65-adda-4f83-8dea-c9583bb6407b_characterizeImage_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/12088e65-adda-4f83-8dea-c9583bb6407b_characterizeImage_1039909_18.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/279b9145-5ddc-4fa9-99f5-1f621215aea6_isr_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/279b9145-5ddc-4fa9-99f5-1f621215aea6_isr_1039909_18.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d86df50c22c5ebafa82f58bdcde76793 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 95cec5877e3d6515a56bd235702edfc9 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9737aab4-d385-4748-9c34-40eb24876526_characterizeImage_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9737aab4-d385-4748-9c34-40eb24876526_characterizeImage_1039908_24.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c9375631-82eb-44e4-ad73-a14be13d7cbb_isr_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c9375631-82eb-44e4-ad73-a14be13d7cbb_isr_1039908_24.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afdec505e727e85a840c2c0a5965bb3a with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 025636b908ed25ab3646e17c00c1e48b parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a8ff4a5a-7ec5-4ed6-a4e5-fb8ca0af81e0_characterizeImage_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a8ff4a5a-7ec5-4ed6-a4e5-fb8ca0af81e0_characterizeImage_1039909_41.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8cc55a36-04b5-4ef6-a1d3-e478e8a81499_isr_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/8cc55a36-04b5-4ef6-a1d3-e478e8a81499_isr_1039909_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:43 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4960714afff7c451ca7134cb95e9d1c1 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash cf0259d21c2a4eab50bcc91a0e736433 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f8940dfd-73ad-482e-9e1c-9eeed2700920_characterizeImage_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f8940dfd-73ad-482e-9e1c-9eeed2700920_characterizeImage_1039909_32.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/0161cb31-34ff-4f96-979d-3139130927ad_isr_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/0161cb31-34ff-4f96-979d-3139130927ad_isr_1039909_32.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e7e41ccb0054c2a5f79b5046b9ceea7 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 2da8d7c728bfc322fced537fd4c5b2cf parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e9e61741-86e5-48e8-a943-dd56e149fd3b_characterizeImage_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e9e61741-86e5-48e8-a943-dd56e149fd3b_characterizeImage_1039909_28.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/184ad2c8-64c8-4a79-9fc3-ef3ddd957080_isr_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/184ad2c8-64c8-4a79-9fc3-ef3ddd957080_isr_1039909_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:43 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c76d90434470c8570ab7ab07bf432f4 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash b9e22ed1897882b430e30b59d0ffd118 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9a24d5a1-627c-4120-973e-9948c7731295_characterizeImage_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9a24d5a1-627c-4120-973e-9948c7731295_characterizeImage_1040013_29.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/4ece4dc6-38e9-4472-8aa4-ea661c213807_isr_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/4ece4dc6-38e9-4472-8aa4-ea661c213807_isr_1040013_29.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10dafb8745d4c0cf9fb792dbbc7519fa with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 8f1040a9cb169a35e26cb83b27a28337 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/a523ad3e-5ea3-40ed-be18-3c8664d1dc5b_characterizeImage_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/a523ad3e-5ea3-40ed-be18-3c8664d1dc5b_characterizeImage_1039907_26.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e3f8dd81-1c2d-4608-b649-899010da8b52_isr_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e3f8dd81-1c2d-4608-b649-899010da8b52_isr_1039907_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:44 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:44 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6793193a8c78be350d3a12d80b993cca with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 987ec4319ffb04c1e481e65bdc93df4f parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0913cea5-1c18-4d7c-b34e-516734e69ed1_characterizeImage_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0913cea5-1c18-4d7c-b34e-516734e69ed1_characterizeImage_1040013_13.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3bcc0839-adaa-44ee-ac45-3d9190865c83_isr_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3bcc0839-adaa-44ee-ac45-3d9190865c83_isr_1040013_13.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69ad7c2f08b1e1b2e9b5ecd908b295e7 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash ecc8373de0aec686d63adfb53701b1ba parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5545a2b9-de7e-4e42-93b2-9b9dc8fae9a6_characterizeImage_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5545a2b9-de7e-4e42-93b2-9b9dc8fae9a6_characterizeImage_1039907_15.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/cb0e4619-6c60-41d1-8468-8e506b6515c7_isr_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/cb0e4619-6c60-41d1-8468-8e506b6515c7_isr_1039907_15.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba2a6ad21c9e939cecc73114d46947fb with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 86b223ff3acaea5292c171ee89e1b047 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/09eb3e31-7811-46f9-a465-48ab58d584f1_characterizeImage_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/09eb3e31-7811-46f9-a465-48ab58d584f1_characterizeImage_1039907_31.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5afcf24b-c67c-459f-81a8-e1646776068d_isr_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5afcf24b-c67c-459f-81a8-e1646776068d_isr_1039907_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:44 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8259550376aee8252751f92d6d2670 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash a961e30e7fedc3acef972a46694a18e0 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/df809a1c-4009-4129-881c-ebbfee8be267_characterizeImage_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/df809a1c-4009-4129-881c-ebbfee8be267_characterizeImage_1039909_10.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3f542260-3b14-48da-bb57-d0823759b580_isr_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3f542260-3b14-48da-bb57-d0823759b580_isr_1039909_10.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228845564c636b514c0ac7517ae77fca with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 88ab133b212e805b293a69a2df1cadf6 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1b768abd-9190-49a8-bf73-b3354fc7ab8c_characterizeImage_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1b768abd-9190-49a8-bf73-b3354fc7ab8c_characterizeImage_1040013_31.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ecad1f25-046d-4d52-9a96-743f6e85a9a3_isr_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/ecad1f25-046d-4d52-9a96-743f6e85a9a3_isr_1040013_31.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2a0fff9b19d82730cbec12d62560645 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 17f43c997fc2c4186f0b2797f49d876d parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/19db0644-a744-4db3-8a3b-d1ad91f566e3_characterizeImage_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/19db0644-a744-4db3-8a3b-d1ad91f566e3_characterizeImage_1039908_44.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/97a5a203-e615-4786-aaad-5effc677902f_isr_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/97a5a203-e615-4786-aaad-5effc677902f_isr_1039908_44.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57a89a68210a64cdcb46fcc5f672d717 with result from task 432 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:44 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash fc724aee0b3ced73940a4f4d118eddf6 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/32fe4d51-1b2a-441c-8555-c591d2770698_characterizeImage_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/32fe4d51-1b2a-441c-8555-c591d2770698_characterizeImage_1039908_28.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8e3f130e-523c-41c4-9368-b1f8ca00fe36_isr_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8e3f130e-523c-41c4-9368-b1f8ca00fe36_isr_1039908_28.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e592b717e9da7eb37dfd1deeff81032f with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 80c31c2f483a4bef4eda47690fb4fb98 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/feddbed3-b9c3-4f79-8271-9476325a7b7c_characterizeImage_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/feddbed3-b9c3-4f79-8271-9476325a7b7c_characterizeImage_1040013_23.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5f15ce7a-ca81-4f49-a46b-317c09673257_isr_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5f15ce7a-ca81-4f49-a46b-317c09673257_isr_1040013_23.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1983bca8623b9c4f542cb58fbff7bf67 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 005b0235072d8148aeddeddee769784b parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/4f4cf4e5-0a1a-4ce5-b941-697e6dd6df7c_characterizeImage_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/4f4cf4e5-0a1a-4ce5-b941-697e6dd6df7c_characterizeImage_1039908_52.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/349a1fd9-76f5-4238-b8ec-cfe2db290aa6_isr_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/349a1fd9-76f5-4238-b8ec-cfe2db290aa6_isr_1039908_52.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b38e67d6cb653f2d5a999fd3e478099c with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 1d404ac56b7c09fcdfadf41a89e7378e parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/bc160d69-7285-45c2-8ded-f264af90e571_characterizeImage_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/bc160d69-7285-45c2-8ded-f264af90e571_characterizeImage_1040013_52.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/598ab02c-80e5-4205-8f88-06de549ea412_isr_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/598ab02c-80e5-4205-8f88-06de549ea412_isr_1040013_52.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d57c28d03a1d2efb1978eb62ecb4cf95 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 504ef47ce77eb3868f4bd6b4c98bd17f parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/136492fd-b853-40fd-a029-28dfbfc5718c_characterizeImage_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/136492fd-b853-40fd-a029-28dfbfc5718c_characterizeImage_1039907_40.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2255ee3-0b90-4f27-9ec1-a3f3f1de7339_isr_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e2255ee3-0b90-4f27-9ec1-a3f3f1de7339_isr_1039907_40.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb6b6690bc7c8e6b0c27683dd4fdeb03 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 134f1020310f61607d5e295b0100d5d8 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b0f0f6ce-72b4-4546-930b-bde349fc640b_characterizeImage_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b0f0f6ce-72b4-4546-930b-bde349fc640b_characterizeImage_1039907_44.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/62ff6f7c-ed0a-46ab-a3c2-b05ccc5c9e8f_isr_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/62ff6f7c-ed0a-46ab-a3c2-b05ccc5c9e8f_isr_1039907_44.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcf60dcddb484d4ebeec3a7b828e7971 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 9df015751f1b438e42ba7bb02e240caa parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/77e8a355-bb3b-46a5-80de-cf453d59ab6a_characterizeImage_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/77e8a355-bb3b-46a5-80de-cf453d59ab6a_characterizeImage_1039907_11.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/52544494-67b4-4e06-a34d-e2928c8d4c5d_isr_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/52544494-67b4-4e06-a34d-e2928c8d4c5d_isr_1039907_11.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d68c4edcc5432a0f6f257d1f2a2bcc with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 585c6ba1ac926456faebcdc5ff20db0b parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b394650d-db59-47d5-a877-191b956f0cfa_characterizeImage_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b394650d-db59-47d5-a877-191b956f0cfa_characterizeImage_1039907_6.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/3ebf76c9-060b-43bd-b026-dd703c77c671_isr_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/3ebf76c9-060b-43bd-b026-dd703c77c671_isr_1039907_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:45 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c09584b37029f6f22702cb707242a87 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 1002a05adbfd25f1d6a628bc591b7944 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a8027259-521d-42d8-abcb-dab758ad4ad5_characterizeImage_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a8027259-521d-42d8-abcb-dab758ad4ad5_characterizeImage_1040013_30.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/80267dec-4152-4b89-b259-bf7393b4f833_isr_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/80267dec-4152-4b89-b259-bf7393b4f833_isr_1040013_30.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71b18788971d31759cf5ada06e465b01 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 31ee9e4bd65be9ffe300ba84161ab601 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/231559c5-5b1f-40cf-a107-b4a4cfc216f4_isr_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/231559c5-5b1f-40cf-a107-b4a4cfc216f4_isr_1039908_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:45 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c5f10e3946f8f21d768fbe737f7c2b with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 67bc41663fbba0fb51e4c923e633bea8 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6cc5a5a7-e7f2-4a12-8520-55869e35caad_characterizeImage_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6cc5a5a7-e7f2-4a12-8520-55869e35caad_characterizeImage_1039908_22.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/843cf252-293d-46eb-887d-eb064f1cf53d_isr_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/843cf252-293d-46eb-887d-eb064f1cf53d_isr_1039908_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:45 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1da0e2efe0458ac056daef6d07d6e14a with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 04a27200a8423c8119a72236cd55ee13 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ed7506e9-4b3b-4482-9b2d-e970976c2cf2_characterizeImage_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ed7506e9-4b3b-4482-9b2d-e970976c2cf2_characterizeImage_1040013_3.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7dff3c35-b4c6-41c9-9476-911a26ac1652_isr_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7dff3c35-b4c6-41c9-9476-911a26ac1652_isr_1040013_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:46 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5866f7432f05d4038342f3454f99010f with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 2516c31fece08c0a2f975ef8a12cdba5 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0f516198-cb5a-46eb-81af-1f11f3f596da_characterizeImage_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0f516198-cb5a-46eb-81af-1f11f3f596da_characterizeImage_1039909_35.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/14494a90-16fa-4b7f-ac9b-e4ebd76ce0a2_isr_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/14494a90-16fa-4b7f-ac9b-e4ebd76ce0a2_isr_1039909_35.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0cb471ff3a2bae23ca4583e633acfed with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash ca36a1ecafc03c39577124341f9bc89a parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/91becfbe-28ba-48d9-a02d-507b840f5ed7_characterizeImage_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/91becfbe-28ba-48d9-a02d-507b840f5ed7_characterizeImage_1039909_50.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/75bdef96-ee09-4196-81b9-2a2c8c576fb5_isr_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/75bdef96-ee09-4196-81b9-2a2c8c576fb5_isr_1039909_50.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431f3bf53454a1488d09ff8f1bf7625f with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 1c23a70df0d08271b435119beb066dfb parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aff54db2-5392-4122-9d92-753e25bfadf7_characterizeImage_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aff54db2-5392-4122-9d92-753e25bfadf7_characterizeImage_1040013_55.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3ed729bb-77f8-40d1-9745-7bc1387e9d64_isr_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/3ed729bb-77f8-40d1-9745-7bc1387e9d64_isr_1040013_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:46 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:46 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:47 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d7b1d390f47ed4a106d01ee2aa991b with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash fabd3c8c35e3868dde6d709df7c8a5fc parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b15ed4a-2fa7-41bf-9c81-e88c0f1a1f2f_characterizeImage_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b15ed4a-2fa7-41bf-9c81-e88c0f1a1f2f_characterizeImage_1039909_59.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/576218f5-a218-4693-b59c-1635f92e0fc6_isr_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/576218f5-a218-4693-b59c-1635f92e0fc6_isr_1039909_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:47 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:48 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46a0944bc902ef6f3f556ad7f2babed2 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 99a2f78a03c96271bd5d0f52fe1efebd parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6872601b-2e3e-4982-9b6c-abe4ec184ce1_characterizeImage_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6872601b-2e3e-4982-9b6c-abe4ec184ce1_characterizeImage_1039907_36.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85942555-848a-4360-a607-dea350b28948_isr_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85942555-848a-4360-a607-dea350b28948_isr_1039907_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:48 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:49 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:49 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:49 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:50 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:50 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:50 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:50 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:51 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:51 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:51 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:52 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:52 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd2ba18793033b70b4aee4f5c73d34f3 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 4429c1a6e9ef6a9e247fac4cefa5d812 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/78e639c5-3eda-47f2-a9d1-29a81e6056ff_characterizeImage_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/78e639c5-3eda-47f2-a9d1-29a81e6056ff_characterizeImage_1039908_50.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/18d54490-e8ca-44a0-ab57-9788de0d3074_isr_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/18d54490-e8ca-44a0-ab57-9788de0d3074_isr_1039908_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:52 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:52 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:53 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:53 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:53 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:54 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e838f32b8f5aef23d9370a4ff42f616f with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 796638ea19555088582cdd0449866003 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3a4fef5-e0fe-46ab-9531-77fd24d80b1e_characterizeImage_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3a4fef5-e0fe-46ab-9531-77fd24d80b1e_characterizeImage_1039909_51.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/7ab18d2b-7bf8-4e21-b6d9-990c89295bfa_isr_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/7ab18d2b-7bf8-4e21-b6d9-990c89295bfa_isr_1039909_51.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5eeccd4b9f404c07952401c5ff642a with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 454d3818f94a8aa0edbeca246b2bb98d parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/cee0f2c9-c2b1-4911-8f8c-1b60ec6a6fb8_characterizeImage_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/cee0f2c9-c2b1-4911-8f8c-1b60ec6a6fb8_characterizeImage_1039909_57.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/97631a7e-c8e8-48ae-8ba0-41f2bd9dbfb1_isr_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/97631a7e-c8e8-48ae-8ba0-41f2bd9dbfb1_isr_1039909_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:54 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd55bb3c3bf232cdbc12724fa6681850 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 78ce932db5b002cf3244611439faf2f6 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/dbfc2e8f-6281-48f5-9a11-08aeec3e02ab_characterizeImage_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/dbfc2e8f-6281-48f5-9a11-08aeec3e02ab_characterizeImage_1039909_25.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/58777bce-5e34-486f-ad9e-4fe0e65ef9dd_isr_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/58777bce-5e34-486f-ad9e-4fe0e65ef9dd_isr_1039909_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:55 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d74e7190c19d50f893db2d2d6adc4ec with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 7b63f361fcbfca3e5842e772e5e83e93 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/2e8838ae-e055-4186-82ea-8efef59c6440_characterizeImage_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/2e8838ae-e055-4186-82ea-8efef59c6440_characterizeImage_1040013_14.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fd0a19-2a66-4e57-bbda-bc4c115e29ed_isr_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fd0a19-2a66-4e57-bbda-bc4c115e29ed_isr_1040013_14.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21346d2410b0deff8f61712576573b35 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 7ac478422cd81bbcafbda2d4d10d6854 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c819e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/8b2f16e2-eee2-465e-bfd3-de0a6f177003_characterizeImage_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/8b2f16e2-eee2-465e-bfd3-de0a6f177003_characterizeImage_1039907_50.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d531925a-9b4b-4c7a-8a6b-7910ce33a925_isr_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/d531925a-9b4b-4c7a-8a6b-7910ce33a925_isr_1039907_50.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de6a9914e2ea8211cf764793f543cc20 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash b3c4e87d1389db9d572aa1969161657d parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b798afc-c195-43cd-9978-93e63c6053c9_characterizeImage_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b798afc-c195-43cd-9978-93e63c6053c9_characterizeImage_1039907_56.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b7df1e4e-db75-4df8-8098-65fea8ee4d7d_isr_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b7df1e4e-db75-4df8-8098-65fea8ee4d7d_isr_1039907_56.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8709883f1ab78cb9279851bd5767440 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash e8002af7f51e506daf7ffc1acb41502e parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c523e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/487cb2b4-07a2-46c2-adfb-077fe2e3c229_characterizeImage_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/487cb2b4-07a2-46c2-adfb-077fe2e3c229_characterizeImage_1039907_25.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ee41a8bd-429c-4655-b702-234c0c1bf1e4_isr_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ee41a8bd-429c-4655-b702-234c0c1bf1e4_isr_1039907_25.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89cb5bed119a9b5efde5326363357911 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash ff7d6e8a00890d724fa031165d4d6e2e parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96e4b657-374d-4ee3-8302-ea89b4ca532d_characterizeImage_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96e4b657-374d-4ee3-8302-ea89b4ca532d_characterizeImage_1040013_9.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/49ec6837-36d8-4de8-9a39-f3e3f3ff15b1_isr_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/49ec6837-36d8-4de8-9a39-f3e3f3ff15b1_isr_1040013_9.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4611109f61e31ca1416f638e6c7933d0 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 2b8face214aaf06c539c6eadeded7df4 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1035e7c8-dc13-45a8-bbc1-a7e2616791b3_characterizeImage_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1035e7c8-dc13-45a8-bbc1-a7e2616791b3_characterizeImage_1039908_39.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c513cbcb-6785-492c-b364-3d47f068bba6_isr_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c513cbcb-6785-492c-b364-3d47f068bba6_isr_1039908_39.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d26569f11af9addbe7988f8eedeb2f5 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash dfbd5f25a9dc1ee8231d66cc5eba9e5e parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6245c8d7-c425-4fcb-a585-0134d102e070_characterizeImage_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6245c8d7-c425-4fcb-a585-0134d102e070_characterizeImage_1039907_14.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a114faf6-fff9-4669-8272-3bb4519c31ac_isr_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a114faf6-fff9-4669-8272-3bb4519c31ac_isr_1039907_14.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f16f3c50b99893648b614cfddb43ac30 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 6ebea1dbfface0338b70965132c2087e parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f78dd498-0816-44f9-a562-6949294b99eb_characterizeImage_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f78dd498-0816-44f9-a562-6949294b99eb_characterizeImage_1039909_17.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1e8c9e7b-7e09-4065-8a62-499c25c3f22c_isr_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1e8c9e7b-7e09-4065-8a62-499c25c3f22c_isr_1039909_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:55 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35549484335cc503268ac9b1411f8f8f with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 182aaa8762f8509d26d252fcab8ea7af parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c813a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bd12a380-450b-4338-97f2-e6285d54372c_characterizeImage_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bd12a380-450b-4338-97f2-e6285d54372c_characterizeImage_1039907_9.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8c752df7-1ede-4d8a-9696-8bbdd01c28b1_isr_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8c752df7-1ede-4d8a-9696-8bbdd01c28b1_isr_1039907_9.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21e43d6dcf2521079a9ed5c011ef71ea with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 7d474f9f636a48ea6211c1dc889d7c76 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c832e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/fcc8618b-b0a9-4426-8e40-770248bf11d8_characterizeImage_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/fcc8618b-b0a9-4426-8e40-770248bf11d8_characterizeImage_1039909_42.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b58575d6-c1a7-4543-b82e-7e2a6c68b327_isr_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b58575d6-c1a7-4543-b82e-7e2a6c68b327_isr_1039909_42.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a52af33e407c85e7a18e898b3ca44628 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 352ba78f6d9206551cef49c3c007cf22 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/e0897aac-5146-4bb7-821e-a409d14452aa_characterizeImage_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/e0897aac-5146-4bb7-821e-a409d14452aa_characterizeImage_1039908_45.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/407c197f-0535-427b-8512-580e4d0d493a_isr_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/407c197f-0535-427b-8512-580e4d0d493a_isr_1039908_45.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a59317917ba91d72da529f2f3c0af349 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 3bbc0410563ffc2caabd9e8e2a7ff48d parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96d91338-ba85-4357-b343-a7dab8e2a5f3_characterizeImage_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96d91338-ba85-4357-b343-a7dab8e2a5f3_characterizeImage_1040013_43.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d2c98e0c-6ea8-4945-88ac-4e9b468fe300_isr_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d2c98e0c-6ea8-4945-88ac-4e9b468fe300_isr_1040013_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:55 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eae6239d7744b565afbf7726a217ae4 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 9acf9516974063589efad2504b143eb1 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c531a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6ba6ceb9-d7e3-4015-8525-ef3126079739_characterizeImage_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6ba6ceb9-d7e3-4015-8525-ef3126079739_characterizeImage_1039909_23.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3853a485-fbba-4abb-993c-223853e4af33_isr_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3853a485-fbba-4abb-993c-223853e4af33_isr_1039909_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:56 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:56 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:56 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8f0de5bad3c13f895e7ac204055a8a with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 9909a22cbee9b5823ea5c33cb46df544 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d0605ef0-adef-4d7c-a134-0be1d4a85c37_characterizeImage_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d0605ef0-adef-4d7c-a134-0be1d4a85c37_characterizeImage_1039909_3.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/508285a5-7b61-421f-ab0d-47c6fd09c434_isr_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/508285a5-7b61-421f-ab0d-47c6fd09c434_isr_1039909_3.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df085e50737540f0731db1f6c06f4d2e with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 4c1f5887ad7fc7d209c90b224e3c7ee0 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7961127e-9cbf-47f5-ac9e-c15c924c21e9_characterizeImage_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7961127e-9cbf-47f5-ac9e-c15c924c21e9_characterizeImage_1039908_42.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/a8593458-ed84-42ad-acc2-05b787fc5f7e_isr_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/a8593458-ed84-42ad-acc2-05b787fc5f7e_isr_1039908_42.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8aa18287cb7e6aca275137cee37bd33 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 7d728cd21c558bb3e6793af41eefa7cc parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c816c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e0537053-33a8-4100-9933-0c2428dc60f9_characterizeImage_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e0537053-33a8-4100-9933-0c2428dc60f9_characterizeImage_1040013_59.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/bb615016-7014-4dfb-8451-efdf0d148d1d_isr_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/bb615016-7014-4dfb-8451-efdf0d148d1d_isr_1040013_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:57 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:57 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:57 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:57 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6999be00f960aedf3f3c5027a5eed9b5 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 11717654afe2a80b1c33fcbdcd1878eb parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0fad67ea-0916-42cb-b89c-67e5d62746b1_characterizeImage_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0fad67ea-0916-42cb-b89c-67e5d62746b1_characterizeImage_1039907_37.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5878d538-50f3-41de-8f60-1f06375f95f7_isr_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/5878d538-50f3-41de-8f60-1f06375f95f7_isr_1039907_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:58 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:58 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 644d3c3bb7ca9dbd03ce87f2fa84421c with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash ac7419c51d49918d91c5d170eb3d4827 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9cfaa779-bc57-4714-9fdd-8d975d34cd7c_characterizeImage_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9cfaa779-bc57-4714-9fdd-8d975d34cd7c_characterizeImage_1040013_33.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1b828630-e93b-40ce-9e9b-f288a45c7f06_isr_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/1b828630-e93b-40ce-9e9b-f288a45c7f06_isr_1040013_33.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7da9bef444b67bc5a4dbf7bf219f24 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 7a09a7a71a185d8749028c4174f81bba parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/761676c7-c97c-4a0a-9786-2527d841b10e_characterizeImage_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/761676c7-c97c-4a0a-9786-2527d841b10e_characterizeImage_1040013_27.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6899b143-5cae-4909-ac38-ba047d493def_isr_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/6899b143-5cae-4909-ac38-ba047d493def_isr_1040013_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:59 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d37f9dd2f631e472b28ee5fd1ada1d2 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash c0ff82641697ba6078fef508101be158 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b119e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3b152bd1-6a5a-4dc6-8865-24cb00896474_characterizeImage_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3b152bd1-6a5a-4dc6-8865-24cb00896474_characterizeImage_1040013_5.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d0e16093-f496-4536-9636-fb2747bca483_isr_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d0e16093-f496-4536-9636-fb2747bca483_isr_1040013_5.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cbd0d16e756123382f167dc17940311 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 6e602a1619198cfecd730c273e517edf parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e465a2a6-2fa6-4203-8227-6819d2acd508_characterizeImage_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e465a2a6-2fa6-4203-8227-6819d2acd508_characterizeImage_1040013_19.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/c1c5a682-cda7-4f09-a7f4-fa92c58b7dee_isr_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/c1c5a682-cda7-4f09-a7f4-fa92c58b7dee_isr_1040013_19.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 124950e10eac098ee9502cc7df90327c with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 60e6a242c022007639d24ffd6345b081 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7bc4eb3c-ecf9-4592-9126-83e3b62d5023_characterizeImage_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7bc4eb3c-ecf9-4592-9126-83e3b62d5023_characterizeImage_1039909_8.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3686448c-4bc4-4dc6-83d1-9d4ff1c78fdc_isr_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/3686448c-4bc4-4dc6-83d1-9d4ff1c78fdc_isr_1039909_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:21:59 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3540bf650afb4c708b1e7fdc6477305e with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 7064eed8598db84179af6f377cb20fcd parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5b9f4e7a-8888-46d2-ada7-83e2f11b9b71_characterizeImage_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5b9f4e7a-8888-46d2-ada7-83e2f11b9b71_characterizeImage_1039909_22.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1dc2f655-6ac6-4251-94bc-4c74aa49e99d_isr_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1dc2f655-6ac6-4251-94bc-4c74aa49e99d_isr_1039909_22.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdb88cdb81682d76811377ed0fc1dde1 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash f546d6373bca0b146e806dcba2f75078 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/49b01d4c-2fa4-4d19-b576-8a9dca7e94f3_characterizeImage_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/49b01d4c-2fa4-4d19-b576-8a9dca7e94f3_characterizeImage_1039907_51.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b99aa1b6-593e-4d2a-9ded-63a956a9aab0_isr_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b99aa1b6-593e-4d2a-9ded-63a956a9aab0_isr_1039907_51.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245834af0f6c8e20064dd97623f87f30 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 2e153a4b5df833bca9be18e2d10d33d3 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1250c0b7-b8a5-4f67-ad8c-b6d76e5611a0_characterizeImage_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1250c0b7-b8a5-4f67-ad8c-b6d76e5611a0_characterizeImage_1039909_53.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/a40817a8-a89c-4963-bfcf-7c6982f458ef_isr_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/a40817a8-a89c-4963-bfcf-7c6982f458ef_isr_1039909_53.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58de5082cca542269d04563770ef93a1 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 4e0099d01ecf9a961c3247c7397761de parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1ee196e8-efe8-4d12-ad2c-ec791a6c5160_characterizeImage_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1ee196e8-efe8-4d12-ad2c-ec791a6c5160_characterizeImage_1040013_15.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8b76ba52-3bdb-4c75-8fa7-d449f658a059_isr_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8b76ba52-3bdb-4c75-8fa7-d449f658a059_isr_1040013_15.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 889133f07f20b4aa0b6e673075dffc49 with result from task 393 parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:00 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 6755e157d5d875c6d7d8b96696687f2d parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ab0b6418-acc7-40d7-a137-7fd2cf3ecb6a_characterizeImage_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ab0b6418-acc7-40d7-a137-7fd2cf3ecb6a_characterizeImage_1039907_53.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a43d219d-8ada-4461-8bad-c9d895ae7102_isr_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a43d219d-8ada-4461-8bad-c9d895ae7102_isr_1039907_53.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 659edfebbb2b55d301bf182ce94883ce with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 92134cee520e99918c9311f4ae4cb032 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/025c9338-f33e-4802-ad5c-68d3934434ab_characterizeImage_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/025c9338-f33e-4802-ad5c-68d3934434ab_characterizeImage_1039909_5.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/20efc07b-ba4e-473e-bc89-4ddaec1d6a3d_isr_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/20efc07b-ba4e-473e-bc89-4ddaec1d6a3d_isr_1039909_5.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c40b984da5ad7d8189d44ed951202ea with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 1de31c2ae1de6476e28a06e0c88b30b7 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0489728b-a24a-41af-a02a-acd22888cf83_characterizeImage_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0489728b-a24a-41af-a02a-acd22888cf83_characterizeImage_1039907_47.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a50d0dcb-122c-45e8-b323-ca4a5d82a4bd_isr_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/a50d0dcb-122c-45e8-b323-ca4a5d82a4bd_isr_1039907_47.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8428cef0de644e0c84e6610855e2a6f0 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash a265fa20182fdaff0a4852192fa5c696 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/99206326-90dd-4260-9f00-09baffa1bdfc_characterizeImage_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/99206326-90dd-4260-9f00-09baffa1bdfc_characterizeImage_1040013_62.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7bf2d1ad-cce4-4c78-8902-0237a4fc806e_isr_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/7bf2d1ad-cce4-4c78-8902-0237a4fc806e_isr_1040013_62.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c6287585edd9126541a22ff7501409 with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 220772dff681aa159f6887468d8baa83 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/9f614824-edda-4e3b-bae8-33a4098778ad_characterizeImage_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/9f614824-edda-4e3b-bae8-33a4098778ad_characterizeImage_1039909_6.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1041e058-b3dd-450e-bb88-7e15a1af15a5_isr_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/1041e058-b3dd-450e-bb88-7e15a1af15a5_isr_1039909_6.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ed341cfd815f2b05faccb1dcbe8611f with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash f8654f406177b7138a07b8448b0727b1 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/7db49775-1cc2-4c66-80b3-0419a6838d7b_characterizeImage_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/7db49775-1cc2-4c66-80b3-0419a6838d7b_characterizeImage_1040013_49.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26b186e0-0652-4a85-8c60-352d8a4441f9_isr_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/26b186e0-0652-4a85-8c60-352d8a4441f9_isr_1040013_49.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e63d6f72e7007824069695ba704b992f with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash e95bc2834e5d92ab7676d8aeb98c90f1 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/941293df-fe04-4026-84a7-b18ed1a745cd_characterizeImage_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/941293df-fe04-4026-84a7-b18ed1a745cd_characterizeImage_1039908_57.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/e38996f9-d49e-466e-abeb-805aea4a61de_isr_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/e38996f9-d49e-466e-abeb-805aea4a61de_isr_1039908_57.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1520998ab13186647bd66564e8ff2d with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 10c3a44a4d24db6c278651563b50835c parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7a23695a-5b01-49fc-a505-ffa692667e4f_characterizeImage_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7a23695a-5b01-49fc-a505-ffa692667e4f_characterizeImage_1039908_10.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/0d770d5f-9eed-4523-9db5-7f7936227465_isr_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/0d770d5f-9eed-4523-9db5-7f7936227465_isr_1039908_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:00 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cce2d0d9388fb9e4a400d3005eb3da2 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash ece3298e63800a1450f2d5bfbdd2ac92 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c8122163-0c1d-40e0-921a-093a1d30e0e0_characterizeImage_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c8122163-0c1d-40e0-921a-093a1d30e0e0_characterizeImage_1039909_38.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/717f5ff3-4cee-43c3-963d-fff5d596bfb8_isr_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/717f5ff3-4cee-43c3-963d-fff5d596bfb8_isr_1039909_38.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3020fbb5b081ba439d08793ab86b359e with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 5246c9814a02f7e08ebb41a562fb9c7f parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87c68071-d84d-4e9b-8b4b-601014b9c657_characterizeImage_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87c68071-d84d-4e9b-8b4b-601014b9c657_characterizeImage_1039907_43.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8f8ac795-e8a5-45f5-a8a6-cbb9d3a01693_isr_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8f8ac795-e8a5-45f5-a8a6-cbb9d3a01693_isr_1039907_43.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c2a3c8dbe392c79986062ca34ce5756 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 057f8e748ddbc6c5cf58aab86017f48f parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/addf8558-fa68-4206-8e20-edebf64d431e_characterizeImage_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/addf8558-fa68-4206-8e20-edebf64d431e_characterizeImage_1040013_53.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9f830c44-c28a-4c6a-b982-dfbb99cc2215_isr_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/9f830c44-c28a-4c6a-b982-dfbb99cc2215_isr_1040013_53.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db985e69adcf3a7bf3cf514cd286689e with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 5b4bea1c273b441d70c046e7938fcc8e parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/43e9f15b-9045-4ddb-a162-5c9be276d10b_characterizeImage_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/43e9f15b-9045-4ddb-a162-5c9be276d10b_characterizeImage_1039907_62.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/1184939a-56b5-447a-9715-8bbbc5e4fa9e_isr_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/1184939a-56b5-447a-9715-8bbbc5e4fa9e_isr_1039907_62.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59ac225f5953aaa32c93b9ff3a5981ae with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 583beed3db74bbaa510a791b39890e89 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df1bc2a4-6a4b-4be9-8bda-d2a7a7da3728_characterizeImage_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df1bc2a4-6a4b-4be9-8bda-d2a7a7da3728_characterizeImage_1040013_37.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f19c3536-0e90-44e5-84e8-d783914596cb_isr_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f19c3536-0e90-44e5-84e8-d783914596cb_isr_1040013_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:01 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb4d6b59a74656a58061b80286d5002d with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 6d452741045466cfba3b2c8015f5ed6c parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/074ae1fb-e217-4801-ab64-5fd0e7fe6ee1_characterizeImage_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/074ae1fb-e217-4801-ab64-5fd0e7fe6ee1_characterizeImage_1039908_37.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b5bd4477-cf5e-4532-8ab1-986ff89f5493_isr_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/b5bd4477-cf5e-4532-8ab1-986ff89f5493_isr_1039908_37.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f699ff9817e678b07dca59484d00e74b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 9fb9e237f47c9e45f3285b9c227c48a3 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/269e31e1-afb5-4f46-8377-037dd73ecb44_characterizeImage_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/269e31e1-afb5-4f46-8377-037dd73ecb44_characterizeImage_1039907_32.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8dc76cec-ce53-4a25-bec7-57d424134cf5_isr_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8dc76cec-ce53-4a25-bec7-57d424134cf5_isr_1039907_32.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0e06b395f37611bd1a480961d7d811d with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 681191ec70c7d0ae924a9c241899b07c parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aedf62bb-cc1f-4f7b-9c29-fc166aaed26c_characterizeImage_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aedf62bb-cc1f-4f7b-9c29-fc166aaed26c_characterizeImage_1040013_20.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/871dedee-2286-47cd-911a-747241541f0d_isr_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/871dedee-2286-47cd-911a-747241541f0d_isr_1040013_20.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 131ad8949464bd273fab832e21dd4327 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash f8cc19f2af7bc6a261ffc865de64c3a6 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df363eb6-bc8a-4004-bc5e-d59d207f679b_characterizeImage_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df363eb6-bc8a-4004-bc5e-d59d207f679b_characterizeImage_1040013_39.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f1ea8bdd-1d9c-42d3-9c61-21cfb461af08_isr_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/f1ea8bdd-1d9c-42d3-9c61-21cfb461af08_isr_1040013_39.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 067b1f96573b8cc5311d383b6bbdfba3 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 8e054a9c8300adf63c77bc36bb70621c parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/48cc0076-213e-47ac-8d7e-3f10280af503_characterizeImage_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/48cc0076-213e-47ac-8d7e-3f10280af503_characterizeImage_1039907_39.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/07a756d0-14fc-412a-b9ce-3baabead3465_isr_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/07a756d0-14fc-412a-b9ce-3baabead3465_isr_1039907_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:02 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b10f8c3ea604dd6907363168eb670a with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 53e0536c29fddf10fb4d4112dcd081bf parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/602c49b0-6055-4210-a7b1-d2202bebb74f_characterizeImage_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/602c49b0-6055-4210-a7b1-d2202bebb74f_characterizeImage_1040013_48.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/eebb55ae-88d3-4af5-bf6e-4e57affc260d_isr_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/eebb55ae-88d3-4af5-bf6e-4e57affc260d_isr_1040013_48.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccd0660a19c1bc50b2c4fd5060739ce1 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 2ca507bc1c79a8a02edb156c4c2b8f4a parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/64dddf31-b639-40ac-9d33-926bdaf23ddb_characterizeImage_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/64dddf31-b639-40ac-9d33-926bdaf23ddb_characterizeImage_1040013_32.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fc7720-8bf2-4e01-9c78-c0bb679ec5ee_isr_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/32fc7720-8bf2-4e01-9c78-c0bb679ec5ee_isr_1040013_32.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0449ba9431f2fbdd9320159c4d02a65f with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash d6df791aaf7aebadf9ec7ca324c528ad parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/367a03dd-b9cc-46f1-b208-f25d202ea832_characterizeImage_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/367a03dd-b9cc-46f1-b208-f25d202ea832_characterizeImage_1039907_21.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e313ea00-383a-41a3-afb9-07b23113814c_isr_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/e313ea00-383a-41a3-afb9-07b23113814c_isr_1039907_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:02 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac89255fb1532abb0a3cfbab77eae25c with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 35de1f363418c514a50e3eeb9dc51299 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/9e84c40d-e755-4364-98fc-fdfaf25ca59b_characterizeImage_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/9e84c40d-e755-4364-98fc-fdfaf25ca59b_characterizeImage_1039907_20.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce616ff0-2fd4-4113-b25d-e4f742d38556_isr_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ce616ff0-2fd4-4113-b25d-e4f742d38556_isr_1039907_20.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09348c7ac08e42feacb3a6dfc129ca26 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 191ee72eaf32ddced6f34c7add36e24c parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/eeacf325-9215-4a2d-b754-674a23d3bde6_characterizeImage_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/eeacf325-9215-4a2d-b754-674a23d3bde6_characterizeImage_1039908_20.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cf0bec3e-e325-41f7-88d3-57799ef4c5dd_isr_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/cf0bec3e-e325-41f7-88d3-57799ef4c5dd_isr_1039908_20.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87e927fa1f17fbf304eb2917f9e1f8d3 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 99f23877dd42f2756a5bcd98e1fb3f39 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/c3e62604-6007-41ef-94d7-ada148e043c6_characterizeImage_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/c3e62604-6007-41ef-94d7-ada148e043c6_characterizeImage_1040013_10.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d283f1dd-ba1e-4544-9b4e-bf7a0599dc49_isr_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/d283f1dd-ba1e-4544-9b4e-bf7a0599dc49_isr_1040013_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:03 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e00f43bec0eae29d18eef63cb18309 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 89142d4f7b908ae48f4385e5fe1badd2 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b105e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/115632bb-2b61-41ea-ae5d-9bff461eb599_characterizeImage_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/115632bb-2b61-41ea-ae5d-9bff461eb599_characterizeImage_1039909_13.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/09d40446-8e50-4e88-98ef-d24582cde1c3_isr_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/09d40446-8e50-4e88-98ef-d24582cde1c3_isr_1039909_13.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ad21bbc1e6c78244ab6dd12a56d96f with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 5148bd3b95719bb07819cf18d75d4909 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/abeb1280-b946-4371-ab5c-62e1ea84e077_characterizeImage_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/abeb1280-b946-4371-ab5c-62e1ea84e077_characterizeImage_1039909_48.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2080addf-f795-42b8-b4f3-b6b8759d77f4_isr_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2080addf-f795-42b8-b4f3-b6b8759d77f4_isr_1039909_48.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40ac9a31f173ae14e4fb539bd167b96b with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 8cdfa07104e8985c127b4f603a540eb7 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6addd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fd11efac-2d72-4f9f-ac2c-65853fb4a439_characterizeImage_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fd11efac-2d72-4f9f-ac2c-65853fb4a439_characterizeImage_1039908_7.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c5acdaa1-f12d-4cfe-8aba-704396146dbb_isr_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c5acdaa1-f12d-4cfe-8aba-704396146dbb_isr_1039908_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 PD 17195155 PD 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c238499c6096371b0cb51dbfb49a09 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash fb9af39ecaef2ad7ffc5946880618daf parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/db2ff676-9a65-43d7-a31a-dcd167b07dd1_characterizeImage_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/db2ff676-9a65-43d7-a31a-dcd167b07dd1_characterizeImage_1039907_30.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c203b9b1-5713-48f9-aa6c-f0da3f7cc633_isr_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/c203b9b1-5713-48f9-aa6c-f0da3f7cc633_isr_1039907_30.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a383590ca19748f66ee89ca1464a8e3 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash f5c8f3736de7f35d8c1ba39962b986e8 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/45b7ce20-7bab-41ba-b0df-56057794112c_characterizeImage_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/45b7ce20-7bab-41ba-b0df-56057794112c_characterizeImage_1039909_4.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9354e3ed-0d87-497d-8814-5ecb6c206a1d_isr_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/9354e3ed-0d87-497d-8814-5ecb6c206a1d_isr_1039909_4.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b833df0a43455a886f00a524c336816 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 233e2c3b211d089f92e315d3f4d173e1 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b123e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7fe8bc3f-e087-4e5f-b28c-b8d626c9ef74_characterizeImage_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7fe8bc3f-e087-4e5f-b28c-b8d626c9ef74_characterizeImage_1039908_26.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/49ae7fe2-094b-47d7-9b51-094c9b5adfba_isr_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/49ae7fe2-094b-47d7-9b51-094c9b5adfba_isr_1039908_26.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90da23f0bee31fd829ea2cd3c06311c4 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash eca8f3afe1111e6373f1e9e1d47ec12a parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b102c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/fe723beb-c325-40f8-be55-647bb55c7341_characterizeImage_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/fe723beb-c325-40f8-be55-647bb55c7341_characterizeImage_1040013_36.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5a64a550-cb67-4801-be76-761aa35785f9_isr_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/5a64a550-cb67-4801-be76-761aa35785f9_isr_1040013_36.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 679c6f2270355370ca6f1d6d6aba1d18 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash b9865d269c4e11928c8c214a243aabd5 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bb86783b-5d81-42f9-aefd-6e8fb3b9c27f_characterizeImage_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bb86783b-5d81-42f9-aefd-6e8fb3b9c27f_characterizeImage_1039907_58.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85605bd4-4e00-4730-84d4-1ea80203341c_isr_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/85605bd4-4e00-4730-84d4-1ea80203341c_isr_1039907_58.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35ff379b3d5679d64df380b4a4943f5e with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 8d9e3db6f98a2cd3b9977ab8a172fb15 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5145ce2c-2d6a-49b0-a253-06bb87404bc4_characterizeImage_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5145ce2c-2d6a-49b0-a253-06bb87404bc4_characterizeImage_1039908_30.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/82ad1849-a1ab-4446-a0ef-e14e22210b7a_isr_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/82ad1849-a1ab-4446-a0ef-e14e22210b7a_isr_1039908_30.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1e5c93f4222f5f262d00a50e35ae16a with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash fceffe08ccdffc7592b30f4e85118278 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/df522365-2582-433a-b08b-67293fb8a007_characterizeImage_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/df522365-2582-433a-b08b-67293fb8a007_characterizeImage_1039908_59.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9542b44e-67d3-4d40-b8cd-31b1d9377bab_isr_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/9542b44e-67d3-4d40-b8cd-31b1d9377bab_isr_1039908_59.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829de2877f38282e4a113e4f65d7ae93 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 1c904c9e2c9ec39b532466609f0a24da parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f86450b6-e3ac-468f-9ddc-67d8bae23761_characterizeImage_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f86450b6-e3ac-468f-9ddc-67d8bae23761_characterizeImage_1039909_60.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b5f87dab-9dfe-4cd5-9581-e01aefad8715_isr_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/b5f87dab-9dfe-4cd5-9581-e01aefad8715_isr_1039909_60.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93a34c85483f1700823e1290cbec61cf with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 1add067a091974e751120721bd8bba22 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/760952bd-a64e-4a60-ba3d-5f7d92a4fe63_characterizeImage_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/760952bd-a64e-4a60-ba3d-5f7d92a4fe63_characterizeImage_1039907_19.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8e2f943a-ce27-4b10-bb15-5fa837e3d604_isr_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/8e2f943a-ce27-4b10-bb15-5fa837e3d604_isr_1039907_19.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 60/2 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 0 failed parsl.dataflow.dflow INFO: Task 166 marked for retry parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 194db5cd5efd7c9cb5f3c6c310c6d8f8 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 1 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f5413695b5cf6928b88f44dabed945 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash ad339f9ebd2d3d771fc0f97623de90c0 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75b079e1-9a5b-47f9-8af9-88b87960d674_characterizeImage_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75b079e1-9a5b-47f9-8af9-88b87960d674_characterizeImage_1039908_19.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f5c95187-09f1-4211-b1be-6b13ae59069d_isr_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/f5c95187-09f1-4211-b1be-6b13ae59069d_isr_1039908_19.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ecb386c9c710c70db7ff26247fce3de with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash d658ae630c71a6c71a2a3faaa82674be parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6e8abb94-9e84-4830-81d7-cf11f0c9bec5_characterizeImage_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6e8abb94-9e84-4830-81d7-cf11f0c9bec5_characterizeImage_1039908_21.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6a3576e9-11c8-48aa-8830-d0b272b0a9fb_isr_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6a3576e9-11c8-48aa-8830-d0b272b0a9fb_isr_1039908_21.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7593581bb6410dbadfe5f7e35bba02b4 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 96e64c5e2e6ec7c3e71ecb2e7228e08c parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b136a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/faab8337-f805-444d-b32d-30d874f6f827_characterizeImage_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/faab8337-f805-444d-b32d-30d874f6f827_characterizeImage_1039908_51.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6c144fd8-852c-49eb-ade7-a74dbc378754_isr_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/6c144fd8-852c-49eb-ade7-a74dbc378754_isr_1039908_51.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 015e63fdf5200fbe9a89a7adf18b39b3 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 24c528a7a2f1c312ef68fa0068dde0e1 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6addda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/fda35c94-42a1-4e36-a85f-d635692c7fb2_characterizeImage_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/fda35c94-42a1-4e36-a85f-d635692c7fb2_characterizeImage_1039907_5.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/add0d38f-aca5-4262-86e3-bd58f290e44d_isr_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/add0d38f-aca5-4262-86e3-bd58f290e44d_isr_1039907_5.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feb26a18421198d7fa470c0c37225a3b with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 7c84c7d2a757283687682ffbceb0b147 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/79815dbc-d4e8-49ce-a10e-b69ff96f33de_characterizeImage_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/79815dbc-d4e8-49ce-a10e-b69ff96f33de_characterizeImage_1039907_46.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9db59b54-40f7-4851-8db4-9f848a92f548_isr_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/9db59b54-40f7-4851-8db4-9f848a92f548_isr_1039907_46.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 437c4c6f397884c6f6ddd242ff658ef0 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash c5fd12fd369efcc178a2b836b7db0f7d parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/92ad5455-1fdc-48e1-a377-6412d0f61f8e_characterizeImage_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/92ad5455-1fdc-48e1-a377-6412d0f61f8e_characterizeImage_1039909_34.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5711b65d-944e-4c97-8a0a-4b4eb1c02a86_isr_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/5711b65d-944e-4c97-8a0a-4b4eb1c02a86_isr_1039909_34.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b197b2f752aed9dd2c967ad2412b50e0 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 3a89bb352c5706ccdbbf97340a768944 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/86850980-a501-4264-ae52-ddc227b6794b_characterizeImage_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/86850980-a501-4264-ae52-ddc227b6794b_characterizeImage_1039908_46.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/839f8c6a-44d6-45dc-89df-cec38e2e781b_isr_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/839f8c6a-44d6-45dc-89df-cec38e2e781b_isr_1039908_46.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 676947f6ba52922a4079beb84f5e746b with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash c3e278e86d97641d88f2bb2272a60599 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1a420601-9236-4097-b7d9-306a2abad849_characterizeImage_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1a420601-9236-4097-b7d9-306a2abad849_characterizeImage_1039907_27.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/034d3022-adfc-4ee4-8b8d-ababc5e3578c_isr_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/034d3022-adfc-4ee4-8b8d-ababc5e3578c_isr_1039907_27.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea28ff86cc43c8efc2ce501242932e99 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 87019cae20985773522f0eae8800a0d3 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d9f8b0cb-4da7-44a9-b125-c13f5face8c0_characterizeImage_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d9f8b0cb-4da7-44a9-b125-c13f5face8c0_characterizeImage_1039908_62.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d4015d94-e462-4138-b6d1-557dac0634ef_isr_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d4015d94-e462-4138-b6d1-557dac0634ef_isr_1039908_62.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bde3d387619459df45c5415bb4294f24 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 2309b1547ac1e707211bb56c7a6350e3 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9d7b1954-54b0-42d8-8d38-7ae5af529e52_characterizeImage_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9d7b1954-54b0-42d8-8d38-7ae5af529e52_characterizeImage_1039908_6.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d9e1ae79-b442-481a-b873-65275a723e47_isr_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d9e1ae79-b442-481a-b873-65275a723e47_isr_1039908_6.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 668481be2d4b3cd1ae3c0ce421b57faf with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash a7d3bef2302637d17a63a9c28419625e parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dea17800-d329-44d1-aa2e-82ceba233d51_characterizeImage_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dea17800-d329-44d1-aa2e-82ceba233d51_characterizeImage_1040013_47.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/56fa741b-3bf5-47e8-9682-5c8e253037e7_isr_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/56fa741b-3bf5-47e8-9682-5c8e253037e7_isr_1040013_47.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06fc212dbde4a29986a37e130a4437eb with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash bfb381560dd233ca4630b07867f7f0c0 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/063acb30-78b2-4314-80ce-7f1b4005a592_characterizeImage_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/063acb30-78b2-4314-80ce-7f1b4005a592_characterizeImage_1039909_11.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04d40e86-c3e5-45b5-8196-4150c29ae1d7_isr_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/04d40e86-c3e5-45b5-8196-4150c29ae1d7_isr_1039909_11.stderr 2024-03-18 05:22:13 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4102680498de38bd929ad24faa2ff88 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash ec00d14842d96fc5b436b7ebc0985ada parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/81101fd4-f1db-4ae8-94b3-ac06f90e6c5d_characterizeImage_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/81101fd4-f1db-4ae8-94b3-ac06f90e6c5d_characterizeImage_1040013_45.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/476b069d-f918-490a-b9ff-7ba4f4c24e89_isr_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/476b069d-f918-490a-b9ff-7ba4f4c24e89_isr_1040013_45.stderr 2024-03-18 05:22:13 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 661648f09191a32b68646493fed01e16 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 2aa7ef0bb656e045455766fadc5905c6 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f4cf2294-8e39-40ca-8529-ef0efb2969f7_characterizeImage_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f4cf2294-8e39-40ca-8529-ef0efb2969f7_characterizeImage_1039909_20.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bd36580f-60ea-48d6-affc-9a7c0a176048_isr_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/bd36580f-60ea-48d6-affc-9a7c0a176048_isr_1039909_20.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9a1443e5cae22f86450b874f74a55cc with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 67ebcbb3b6592ba1a62f17df201ca5ed parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/326aa9a4-c048-466f-932e-e04f7f04c3c1_characterizeImage_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/326aa9a4-c048-466f-932e-e04f7f04c3c1_characterizeImage_1039908_48.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d10dee52-a152-4d41-a2f2-ebc815a91cd3_isr_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/d10dee52-a152-4d41-a2f2-ebc815a91cd3_isr_1039908_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:14 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1373a8e115240495a19212790311939 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash bc7223cbc8619315cbce06af2d96f559 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2e2af495-ee7c-4878-8a56-1335d818f567_characterizeImage_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2e2af495-ee7c-4878-8a56-1335d818f567_characterizeImage_1039909_55.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2c8be529-6ed2-4400-b472-c0527cc01f12_isr_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/2c8be529-6ed2-4400-b472-c0527cc01f12_isr_1039909_55.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db3187478e2e29ab810f0f9c3a682858 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash a3eaf651c1d1d23face4c31e70a1a917 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/33ed53c3-0fd1-4779-b50e-eff43f669a58_characterizeImage_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/33ed53c3-0fd1-4779-b50e-eff43f669a58_characterizeImage_1040013_26.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/b7898236-b20f-41ae-bd8e-1398c527b9de_isr_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/b7898236-b20f-41ae-bd8e-1398c527b9de_isr_1040013_26.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06e43a8715a0217d784d3b8c3502d3e6 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 69084ddb1eae5d07176d94ea519ed211 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5218424f-d511-4e72-af40-3f56a001038c_characterizeImage_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5218424f-d511-4e72-af40-3f56a001038c_characterizeImage_1039909_49.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/030a3370-06c8-43d6-a463-a614752b2862_isr_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039909/030a3370-06c8-43d6-a463-a614752b2862_isr_1039909_49.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 487c7509d3939ab6a7fdd479f20fce6b with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 6aa85ed70e2832931e42444fd42a96c9 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/82dc26f9-f453-4aec-803c-e55005c29863_characterizeImage_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/82dc26f9-f453-4aec-803c-e55005c29863_characterizeImage_1040013_42.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2298c951-f1da-4521-bc62-af7c3b9a05d6_isr_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/2298c951-f1da-4521-bc62-af7c3b9a05d6_isr_1040013_42.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7164369d64115f13854e75c47303b30d with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 15183786a8854cf7d602e9f82cb1df98 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/76756253-b440-4733-ae4a-3f4142ad156f_characterizeImage_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/76756253-b440-4733-ae4a-3f4142ad156f_characterizeImage_1039907_17.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/518f0947-af32-4eaf-8aff-795a01cde991_isr_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/518f0947-af32-4eaf-8aff-795a01cde991_isr_1039907_17.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a715d59fdfd13d0c0aa91138cecb978 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 9d870f6fdea6d4cec99ee399734365c5 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5de2c631-0cec-49ec-9a86-fa2a47d73d61_characterizeImage_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5de2c631-0cec-49ec-9a86-fa2a47d73d61_characterizeImage_1040013_7.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8a7a1db5-35d7-4f58-a318-3c00ba113444_isr_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/8a7a1db5-35d7-4f58-a318-3c00ba113444_isr_1040013_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:14 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b1e5f352d7a7f934afe884cc6781feb with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash c29e57b1d70a3ee8429e91d3d2d3408a parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/19a95757-e342-4bdd-9cc6-7e466ef2c3c6_characterizeImage_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/19a95757-e342-4bdd-9cc6-7e466ef2c3c6_characterizeImage_1039907_42.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/54ac5339-1b34-46e6-b903-c89e00b91fef_isr_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/54ac5339-1b34-46e6-b903-c89e00b91fef_isr_1039907_42.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b06f30a2f37067773e562d6e2414a03 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash da2ddb5f457de1a6c544b8dc25083fde parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b139c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5220ad20-16ba-404f-893c-628cceb122de_characterizeImage_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5220ad20-16ba-404f-893c-628cceb122de_characterizeImage_1039908_60.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/73569d78-1f3f-4e1c-a665-47cac99575da_isr_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/73569d78-1f3f-4e1c-a665-47cac99575da_isr_1039908_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:15 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05058943e33d053567d0e9fa1faf360f with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 1ab4fc1fd4e4118feb66d1b70fd6c63c parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/356c51d7-2c8a-42a1-ab67-7a7fc3b4755e_characterizeImage_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/356c51d7-2c8a-42a1-ab67-7a7fc3b4755e_characterizeImage_1039908_53.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c23b73e2-909f-45a6-9624-968189f61a1e_isr_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c23b73e2-909f-45a6-9624-968189f61a1e_isr_1039908_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:15 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f3d450cc9e4e5f43288214b8a96fba5 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 9ca1f149ba597f2ebc5a3439e564a0e5 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/12fcbe7b-21d2-4d06-8b09-4963c28627ce_characterizeImage_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/12fcbe7b-21d2-4d06-8b09-4963c28627ce_characterizeImage_1039907_55.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ef287fef-1a31-408f-b143-2854793ac201_isr_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/ef287fef-1a31-408f-b143-2854793ac201_isr_1039907_55.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e6c864771e823c595a77d11a3f8c54 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 186f279f7d5016d9f6afd10d359d9b82 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87500765-0bb8-49ea-b12c-2635fcd3d518_characterizeImage_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87500765-0bb8-49ea-b12c-2635fcd3d518_characterizeImage_1039907_23.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/dfbaa47f-093f-4513-b3b5-0f918a659edf_isr_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/dfbaa47f-093f-4513-b3b5-0f918a659edf_isr_1039907_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:16 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:17 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4e1a8fc00a74e580dde66d445f0e75 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 47369ecce78dfca9bcfbfa5bb6e0e763 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/902dd33e-97b6-48a0-9980-e5854a8110c1_characterizeImage_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/902dd33e-97b6-48a0-9980-e5854a8110c1_characterizeImage_1039907_24.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f1df871f-771a-4bd9-9da1-e1d684c3b072_isr_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/f1df871f-771a-4bd9-9da1-e1d684c3b072_isr_1039907_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:17 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:17 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad12da07f36b6e6d0d59f2ef3b341048 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 32f47c1d5a416002421be9db6ec07027 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/352e2781-dab4-4819-8945-acd3765ef681_characterizeImage_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/352e2781-dab4-4819-8945-acd3765ef681_characterizeImage_1039907_16.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/834c2b7d-cd23-43c3-8042-8a636b7759b1_isr_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/834c2b7d-cd23-43c3-8042-8a636b7759b1_isr_1039907_16.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a254a83375da60f1f3b37326159d7919 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 40fa87a5c9d2b2fc659187934d9881e6 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/84c0d36f-37eb-40b6-b5c3-01a21cb6314c_characterizeImage_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/84c0d36f-37eb-40b6-b5c3-01a21cb6314c_characterizeImage_1039908_23.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/eaef1ad0-9940-47b5-8574-705ff73291c0_isr_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/eaef1ad0-9940-47b5-8574-705ff73291c0_isr_1039908_23.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0694186a268277180fd4b43c34a7dfd5 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 0200cc6f9f3ca884a0f92641b832ce98 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b936b06-3318-485a-9131-7d511c5a48de_characterizeImage_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b936b06-3318-485a-9131-7d511c5a48de_characterizeImage_1039907_18.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b64e3f85-afd9-40fe-8ebe-36e823d5bf77_isr_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039907/b64e3f85-afd9-40fe-8ebe-36e823d5bf77_isr_1039907_18.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18e702bf10a8e2227dd5617a27b1465 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash badd635e1bfccd55f9975e9018a0706d parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ece9f156-d77d-4665-bd92-56582f57d33b_characterizeImage_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ece9f156-d77d-4665-bd92-56582f57d33b_characterizeImage_1040013_35.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/cfee3e33-5203-437f-aca2-f8c52f103b8c_isr_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1040013/cfee3e33-5203-437f-aca2-f8c52f103b8c_isr_1040013_35.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b7af383bf1d59a352c7f248478dd17a with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 689412462b68fbf8142861fcbcfb567f parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1d439f48-75f2-430e-99c9-d03add0a823a_characterizeImage_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1d439f48-75f2-430e-99c9-d03add0a823a_characterizeImage_1039908_43.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c74a4d33-19a0-4448-b89b-592d779a357b_isr_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/c74a4d33-19a0-4448-b89b-592d779a357b_isr_1039908_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:18 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:18 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc839f4152b89f0a69c568d88ac4aaff with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash b6f83f23dea40359df9040e59ba35718 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/882ad7f4-b117-469a-8686-d1c7c2477c1a_characterizeImage_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/882ad7f4-b117-469a-8686-d1c7c2477c1a_characterizeImage_1039908_32.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8668169c-7ccf-4f21-82d3-0c7c20ef0e9e_isr_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/8668169c-7ccf-4f21-82d3-0c7c20ef0e9e_isr_1039908_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:18 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:19 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:19 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:19 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c6b13d8a59364a90e99b09bf33f5dc with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash c3596bb9244e313cb9194114287b8e64 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75f7f23f-4139-4ed1-84cc-078ec64510fd_characterizeImage_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75f7f23f-4139-4ed1-84cc-078ec64510fd_characterizeImage_1039908_4.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/bb4afa43-1894-4cae-bfa7-34eb1a278211_isr_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/isr/1039908/bb4afa43-1894-4cae-bfa7-34eb1a278211_isr_1039908_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:19 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:20 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:20 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:21 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:21 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:22 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:22 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:23 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:23 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:24 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:24 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:25 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:25 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:25 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:26 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:26 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:27 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:27 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:28 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:28 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:29 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:30 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:31 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 211 try 0 failed parsl.dataflow.dflow INFO: Task 211 marked for retry parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 28ab71a4e222f92ac8fb949aee29f933 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 1 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:32 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:33 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:34 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:35 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:36 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:37 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:38 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:39 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:39 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:40 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:40 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:41 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:41 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69afef8340b0002d8983725f5def3cd5 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 6d7eb5934572301e2b159919866fcbec parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/ee743ba7-9229-40f4-8449-d939952b3d89_calibrate_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/ee743ba7-9229-40f4-8449-d939952b3d89_calibrate_1039908_54.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/de373069-d837-4c3c-a182-2e1845eab271_characterizeImage_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/de373069-d837-4c3c-a182-2e1845eab271_characterizeImage_1039908_54.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 915d8a8d455d4d590534929d9054b3f6 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash b03731cdceff55165c6de6850d342892 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/815d1031-3766-4803-9db4-16ee40a30e3a_calibrate_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/815d1031-3766-4803-9db4-16ee40a30e3a_calibrate_1039907_7.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4610347e-8e58-4aee-8694-496363deff55_characterizeImage_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4610347e-8e58-4aee-8694-496363deff55_characterizeImage_1039907_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:42 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d16b6281b50395e86b3da64446101c39 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash e5add391ee4ced5d4b068fcddfc588a2 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f6924ff7-3fa6-430e-8e42-800ebfc80c78_calibrate_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f6924ff7-3fa6-430e-8e42-800ebfc80c78_calibrate_1039908_1.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1dd4fcee-0627-4bd8-b25d-13f066914dde_characterizeImage_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1dd4fcee-0627-4bd8-b25d-13f066914dde_characterizeImage_1039908_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:42 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:43 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:43 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:44 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00a353909e900d49a6d0991938d50b61 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 1f53d1c1234573da43113def60a0f639 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/454abbfc-4d78-4ca0-941e-eda3f9147a30_calibrate_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/454abbfc-4d78-4ca0-941e-eda3f9147a30_calibrate_1039907_45.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/3d0c9e30-f20c-4c76-82fa-11b5efc06762_characterizeImage_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/3d0c9e30-f20c-4c76-82fa-11b5efc06762_characterizeImage_1039907_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:44 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:44 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 0 failed parsl.dataflow.dflow INFO: Task 154 marked for retry parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 342858d2be62085ffb45a8427efa2341 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 1 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:45 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:45 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8323045890a3c4195531e4bf7beea2d with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 996837020ded87f1f980073bd749b573 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3e29ae7b-f363-4334-bd3a-106e8fbed689_calibrate_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3e29ae7b-f363-4334-bd3a-106e8fbed689_calibrate_1039908_34.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c61ed480-97f4-4df9-90b1-0699c30e21fc_characterizeImage_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c61ed480-97f4-4df9-90b1-0699c30e21fc_characterizeImage_1039908_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:46 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 2024-03-18 05:22:48 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:22:48 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:48 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:49 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:49 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:50 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 331 try 0 failed parsl.dataflow.dflow INFO: Task 331 marked for retry parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash cc87dcef0be0080c7329819e7855e780 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 1 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:50 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:51 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:51 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc70080b30b15a26284df5810ba0d588 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash b108e14ad6ece34a19ac70840c6f5824 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/05f28d4f-058f-4bdb-801e-957ade3dbfc0_calibrate_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/05f28d4f-058f-4bdb-801e-957ade3dbfc0_calibrate_1039909_31.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/17e5ce8d-e7d7-40c3-ad1e-0af57f415f3e_characterizeImage_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/17e5ce8d-e7d7-40c3-ad1e-0af57f415f3e_characterizeImage_1039909_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:52 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c8b65a6b4f85225127133e5df24e33f with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash c2c4de091f477e9b2d3a9e1d36d4df04 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/47ef03b3-91f2-47ac-99f7-de2e50e13517_calibrate_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/47ef03b3-91f2-47ac-99f7-de2e50e13517_calibrate_1039907_49.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b8940c74-6178-4ec8-96ad-ccee02f74370_characterizeImage_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b8940c74-6178-4ec8-96ad-ccee02f74370_characterizeImage_1039907_49.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfbc2ef324735c33f4f1f1d06edb70ca with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 82ec651a99a780f292f7804b8bbd500c parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a903f7f1-c6ac-4de6-bfe9-62fdce0ac36f_calibrate_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a903f7f1-c6ac-4de6-bfe9-62fdce0ac36f_calibrate_1039909_44.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0873391f-8951-432b-8425-1d834c095231_characterizeImage_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0873391f-8951-432b-8425-1d834c095231_characterizeImage_1039909_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:52 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:53 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:53 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:54 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:54 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:54 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 0 failed parsl.dataflow.dflow INFO: Task 226 marked for retry parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 0ca5729918013c6c6251f94dc60ba8c8 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 1 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:55 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:55 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:55 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:56 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:56 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 236b427db1334751b6b11ebdbd983b6e with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash b9b075c08e2f2ad8b237566def113470 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/280c77b8-5061-4df8-a9b9-f2afb6f16f51_calibrate_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/280c77b8-5061-4df8-a9b9-f2afb6f16f51_calibrate_1039909_33.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3945953-7241-42e1-9131-1c1eeee5f936_characterizeImage_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3945953-7241-42e1-9131-1c1eeee5f936_characterizeImage_1039909_33.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 239af51659c0fd29c7b30f8111e668d8 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash cc5f5d45ca342b836cbb7bdc49076b70 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81884a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2621eaa0-bade-4be9-a420-b752e672c6cb_calibrate_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2621eaa0-bade-4be9-a420-b752e672c6cb_calibrate_1039907_13.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/acf09252-27cd-444e-b78e-142dfb93357a_characterizeImage_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/acf09252-27cd-444e-b78e-142dfb93357a_characterizeImage_1039907_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:57 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:57 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:57 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ccea73d8b4daeb2cdc60e5c895d3bc with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 6b28b14a4a9a326243e4db5bf6d48d0d parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/759a60f0-d689-4e41-a2ca-4c188a2cfd6a_calibrate_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/759a60f0-d689-4e41-a2ca-4c188a2cfd6a_calibrate_1039907_52.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f0ac72f0-02b3-4ea6-b0e0-f20e40c36f94_characterizeImage_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f0ac72f0-02b3-4ea6-b0e0-f20e40c36f94_characterizeImage_1039907_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:58 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de784107e4782bf05eaf7517c7278a24 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash b77b6fc02e48a983a39a9fc7f641710e parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ff12cbf9-d445-41df-82c3-8cf4347e3c98_calibrate_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ff12cbf9-d445-41df-82c3-8cf4347e3c98_calibrate_1039909_14.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/81feed20-88f8-4401-8f0b-4e6884fdc37f_characterizeImage_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/81feed20-88f8-4401-8f0b-4e6884fdc37f_characterizeImage_1039909_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:58 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:58 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2ab276980124150c7632c94e74432be with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 815a059dd5e13b4d3c7018d8f9891209 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/df0e0507-d602-4693-b374-ac1e19832b4f_calibrate_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/df0e0507-d602-4693-b374-ac1e19832b4f_calibrate_1039909_43.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/61d52d01-d538-4492-884a-35c3efa7e4df_characterizeImage_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/61d52d01-d538-4492-884a-35c3efa7e4df_characterizeImage_1039909_43.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 807b40ebca627977e1929e2fa504a86c with result from task 142 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:59 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 81c4a5c627e085c7a9cead4b97f4b7cd parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8d2fe28f-92ac-4e50-8272-2fe8c23390b3_calibrate_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8d2fe28f-92ac-4e50-8272-2fe8c23390b3_calibrate_1039908_8.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/162ce5bb-4b06-4b53-a617-df4b24f4a4d1_characterizeImage_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/162ce5bb-4b06-4b53-a617-df4b24f4a4d1_characterizeImage_1039908_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:22:59 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a901406c7351950e593efe4088de667 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash f3b91307ff2b365fa8c47429dbe48be5 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1d8b06be-5f81-4871-9572-9dda8febe43a_calibrate_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1d8b06be-5f81-4871-9572-9dda8febe43a_calibrate_1039907_41.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/062d865d-bec5-4d63-861f-17ca42e8ea88_characterizeImage_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/062d865d-bec5-4d63-861f-17ca42e8ea88_characterizeImage_1039907_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:00 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:00 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac7d1dbf949fc9cfd118813f98f4657b with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash b8058374ab9939795d871bff215db25e parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9a90b2cc-99dc-4406-ade1-c48344bb6c20_calibrate_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9a90b2cc-99dc-4406-ade1-c48344bb6c20_calibrate_1039908_18.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/09bfdbb2-4677-4e9b-9c8c-f35be7ab2a51_characterizeImage_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/09bfdbb2-4677-4e9b-9c8c-f35be7ab2a51_characterizeImage_1039908_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:01 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:01 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:01 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ab8a1a0eb7b55199f7c31c4b23b2be1 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 909ed7e9e2cb9a58c03c0d4128de3b83 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/459cacff-4578-4191-a8c7-0bb147d566e6_calibrate_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/459cacff-4578-4191-a8c7-0bb147d566e6_calibrate_1039907_4.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/892bfcc1-768e-4ded-b208-c03024600cdd_characterizeImage_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/892bfcc1-768e-4ded-b208-c03024600cdd_characterizeImage_1039907_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:02 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:02 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:03 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:03 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228927625c00ee35a3795d772828c73e with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 17caf2ddc8ae0c99f971f67c01e0d257 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81898a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/277bfc1e-3d6d-4fcd-b820-38be7d83cb2f_calibrate_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/277bfc1e-3d6d-4fcd-b820-38be7d83cb2f_calibrate_1039907_22.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/638e7d0d-f723-4729-a983-aec64227bce3_characterizeImage_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/638e7d0d-f723-4729-a983-aec64227bce3_characterizeImage_1039907_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:04 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752b8c5a0d59ef3b02c3ff428a95e3fc with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash eb6b809d75bf61e4722229098585c1b9 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/e1ff9285-3ac5-458b-926e-dabd1ce69611_calibrate_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/e1ff9285-3ac5-458b-926e-dabd1ce69611_calibrate_1039909_40.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b40e686-6722-49a2-8d23-7178afed9c56_characterizeImage_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b40e686-6722-49a2-8d23-7178afed9c56_characterizeImage_1039909_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:04 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:05 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:05 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:05 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:06 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:06 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:07 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:07 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:07 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c78aa95f2f14d2b45573bb61e5f900f with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 8926e40875721033410401e4e291a579 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/aa17032e-1f38-4d23-bd34-e83ca57c9358_calibrate_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/aa17032e-1f38-4d23-bd34-e83ca57c9358_calibrate_1039909_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8a9081dc-7485-4a08-84a7-e578d11c0dce_characterizeImage_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8a9081dc-7485-4a08-84a7-e578d11c0dce_characterizeImage_1039909_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:08 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:08 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3306e4892f2600b3d597fdef97677164 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 46941856b57bc29114d5ee56ea6faf37 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b576b967-0594-48c0-83f8-74131447f21d_calibrate_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b576b967-0594-48c0-83f8-74131447f21d_calibrate_1039908_31.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c9421425-c7d3-4c2d-af8d-5891db14c9f6_characterizeImage_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c9421425-c7d3-4c2d-af8d-5891db14c9f6_characterizeImage_1039908_31.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5481614ac04765c629f5fc8954acebe3 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash f7c7e9f2e92e80b74b67ea35bf159c8b parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/67ac85de-0628-46f0-ad79-25fa41e87892_calibrate_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/67ac85de-0628-46f0-ad79-25fa41e87892_calibrate_1039909_21.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c5831da2-b309-4372-97a8-025866ec0406_characterizeImage_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c5831da2-b309-4372-97a8-025866ec0406_characterizeImage_1039909_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:09 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 272a6cf8fee9fc7c4db4525a7071ba06 with result from task 322 parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:09 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 1599b4c5a40d0717ab3ef63667ce248c parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/07f234c1-5bd9-4d7e-b536-3cb67212ce29_calibrate_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/07f234c1-5bd9-4d7e-b536-3cb67212ce29_calibrate_1039907_59.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/328ade1c-5e0f-4de1-ada3-25946310e3b2_characterizeImage_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/328ade1c-5e0f-4de1-ada3-25946310e3b2_characterizeImage_1039907_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:09 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:10 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:10 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09bd6826be112625c709cd399383d2e with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 4311fe3de4aefde256141a9d9fde5f41 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c8ef4b28-bfce-440e-ba1c-a4185da32d9e_calibrate_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c8ef4b28-bfce-440e-ba1c-a4185da32d9e_calibrate_1039908_14.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c636f9b6-d6e9-4870-a6b2-379a053f85fb_characterizeImage_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/c636f9b6-d6e9-4870-a6b2-379a053f85fb_characterizeImage_1039908_14.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b64622d3aade22ae7b24f04f0943a16 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash de953591d18744186f10618db944e390 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/cdd5d50a-a235-409c-b666-65c03913394a_calibrate_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/cdd5d50a-a235-409c-b666-65c03913394a_calibrate_1039907_48.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ee127dc9-41b9-4df7-bad7-d07e275d6ffd_characterizeImage_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ee127dc9-41b9-4df7-bad7-d07e275d6ffd_characterizeImage_1039907_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:10 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:10 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:11 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b48c15d429cf86b1823387cb2827d7e with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash f70e00e7899c2e45df53a2a99d8db55c parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7908610e-241f-4374-8811-f5941da55e74_calibrate_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7908610e-241f-4374-8811-f5941da55e74_calibrate_1040013_1.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4438386a-b2a7-45fa-a9d8-02ea5d670e61_characterizeImage_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4438386a-b2a7-45fa-a9d8-02ea5d670e61_characterizeImage_1040013_1.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb4eafb93fe1d4849e13433c31fd54e4 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a5739a184c48ecf3bccc5d7a1d15b863 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d3cd410c-879e-4608-bb2f-a849857249e2_calibrate_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d3cd410c-879e-4608-bb2f-a849857249e2_calibrate_1039907_12.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/d297764d-8e64-4a87-9c5a-7fc03f86ede4_characterizeImage_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/d297764d-8e64-4a87-9c5a-7fc03f86ede4_characterizeImage_1039907_12.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cab20add429f3f98728cf8f9bcf798d with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash fbcb6bf4810824e9348522204c4e1015 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4796a652-608e-48b7-9628-537e9dcd9851_calibrate_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4796a652-608e-48b7-9628-537e9dcd9851_calibrate_1039908_16.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dddb28ac-8b86-48cd-b4b5-c6716646745c_characterizeImage_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dddb28ac-8b86-48cd-b4b5-c6716646745c_characterizeImage_1039908_16.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d8f7b13f213b58e673ed87fd3b3a4c8 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 858b0b3ee628f44220b0e73c7b30eef8 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4007fba1-4875-4322-80a2-5e21464eee1b_calibrate_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4007fba1-4875-4322-80a2-5e21464eee1b_calibrate_1039908_3.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fc3a5e9a-2f1c-4667-8059-10c0a260e29b_characterizeImage_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fc3a5e9a-2f1c-4667-8059-10c0a260e29b_characterizeImage_1039908_3.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24384de3379c1ca0276e4b233d9507e4 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 7c438f5bf8c7f4769f774194dcc4cc1d parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1c2e039d-6abf-4928-846a-26e7bd235089_calibrate_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1c2e039d-6abf-4928-846a-26e7bd235089_calibrate_1039908_15.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/33d793df-d729-46dc-9f05-63314a7bcb3d_characterizeImage_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/33d793df-d729-46dc-9f05-63314a7bcb3d_characterizeImage_1039908_15.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb3b3e3726d65abc9edb6099461e7e2a with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash f20ecdfc9f1c767607ea5b8232552bea parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/8dc37be1-ae18-42c6-8b8b-72da68f77d4f_calibrate_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/8dc37be1-ae18-42c6-8b8b-72da68f77d4f_calibrate_1039907_54.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/cebe6ec8-a21d-4f47-ad38-e3f031e63b20_characterizeImage_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/cebe6ec8-a21d-4f47-ad38-e3f031e63b20_characterizeImage_1039907_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:11 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d805df08a8d927396eab9928c7a6b58f with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash ccf3ca7102c5bea8dd020da1d28b1b5c parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2dd02ded-6fab-4faa-9581-8b14151ca26f_calibrate_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2dd02ded-6fab-4faa-9581-8b14151ca26f_calibrate_1039908_33.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/21f739ea-27ce-47ec-82b6-9a8ba94b8b51_characterizeImage_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/21f739ea-27ce-47ec-82b6-9a8ba94b8b51_characterizeImage_1039908_33.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3268ff95746ce7831c37a55dc6cc9c64 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash a83d6c89a9f8f50cd89064c94f1b2ee0 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bdc74dd7-ad31-4d1d-b27b-815b2c7fca32_calibrate_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bdc74dd7-ad31-4d1d-b27b-815b2c7fca32_calibrate_1039908_36.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/af308e70-253e-40e3-b37b-70d1eabccc86_characterizeImage_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/af308e70-253e-40e3-b37b-70d1eabccc86_characterizeImage_1039908_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:12 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:12 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506f787fb195864fafb84a5200fea608 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 7bc073b12f6253fa5de7a0439659ef02 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/615363f0-4deb-4c52-9dbc-6fde8323daed_calibrate_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/615363f0-4deb-4c52-9dbc-6fde8323daed_calibrate_1039907_28.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/e3d11a5b-c083-4789-8cf8-5b7bbe82cd82_characterizeImage_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/e3d11a5b-c083-4789-8cf8-5b7bbe82cd82_characterizeImage_1039907_28.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4cba0e8143075db3527398f1e1c595f with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 610b08c98d9c022703559cf7a95bd30a parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8149a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/e1a287c5-b132-4e50-8a26-ddbe0c0e86b0_calibrate_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/e1a287c5-b132-4e50-8a26-ddbe0c0e86b0_calibrate_1039907_33.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/139b6178-b9a1-403c-97e3-7585ea1b6e28_characterizeImage_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/139b6178-b9a1-403c-97e3-7585ea1b6e28_characterizeImage_1039907_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:13 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d16236045dbf3b1bfd3af464dead73 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash ac9b1ecd26467f56869188de39912d9c parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a722cad6-8868-473a-bb10-58917058f48a_calibrate_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a722cad6-8868-473a-bb10-58917058f48a_calibrate_1039907_34.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/58cc4737-6ae8-4028-a540-cdfb40d51074_characterizeImage_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/58cc4737-6ae8-4028-a540-cdfb40d51074_characterizeImage_1039907_34.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4a76527ad9f5f982207a2d7d60ef11c with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 2aa148237e7e8abe55fe0fc81b2d17c7 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/172d456e-2739-4933-baa3-60bed4d7eb61_calibrate_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/172d456e-2739-4933-baa3-60bed4d7eb61_calibrate_1039908_55.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/81663b88-188c-43f8-9286-61e59bff6a57_characterizeImage_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/81663b88-188c-43f8-9286-61e59bff6a57_characterizeImage_1039908_55.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c0e5125010df529d6b65b2c1352b5a with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash f2d494180da2e23a5072df119240872e parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81699e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/4bf79b47-8f28-4c0c-aacc-c78e2edf01b6_calibrate_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/4bf79b47-8f28-4c0c-aacc-c78e2edf01b6_calibrate_1039907_1.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/2cad68d3-d8eb-4a5e-b90c-c82f4e6b9c67_characterizeImage_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/2cad68d3-d8eb-4a5e-b90c-c82f4e6b9c67_characterizeImage_1039907_1.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebafc9785029a3996b2e3352b4170aa0 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 0ff59b63c1e90e808ac31f21106a41ba parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a4c9c0a0-d89f-40d7-9e43-c9990a031051_calibrate_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a4c9c0a0-d89f-40d7-9e43-c9990a031051_calibrate_1039909_7.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/757a8220-1895-44c6-ba95-af178355b019_characterizeImage_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/757a8220-1895-44c6-ba95-af178355b019_characterizeImage_1039909_7.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 024708ff8a587b87e96582fafd8356f7 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 4d3922946db74ed5a528e3ca9e1a2740 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f111a4d1-23cc-4a78-9906-7c373dda5c12_calibrate_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f111a4d1-23cc-4a78-9906-7c373dda5c12_calibrate_1039908_17.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0af03258-01cb-42a1-9c2d-59613b354cd5_characterizeImage_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0af03258-01cb-42a1-9c2d-59613b354cd5_characterizeImage_1039908_17.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 343 try 0 failed parsl.dataflow.dflow INFO: Task 343 marked for retry parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 03d8ac9af6680731de874090147ac7ae parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 1 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 526 try 0 failed parsl.dataflow.dflow INFO: Task 526 marked for retry parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 31ee9e4bd65be9ffe300ba84161ab601 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 1 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stderr 2024-03-18 05:23:18 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:23:18 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:19 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:19 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:19 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:20 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:20 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb2666e2910a795c9008e4b4bdbcab11 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 91caa66fc3160a644afa0d99d353607a parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8189260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4126f252-8523-4617-8090-c9736892b27e_calibrate_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4126f252-8523-4617-8090-c9736892b27e_calibrate_1040013_28.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3f69c5b0-101f-4efb-bffa-f63c1e84a2d8_characterizeImage_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3f69c5b0-101f-4efb-bffa-f63c1e84a2d8_characterizeImage_1040013_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:21 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:21 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80955f6c0dcf67f8100aae1c122b7a95 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 6ac7cd78f270b2e66ca393d06fbeafcb parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/4b60c4dd-7931-438b-8841-698bea5fa028_calibrate_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/4b60c4dd-7931-438b-8841-698bea5fa028_calibrate_1039909_54.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f12a8d9-cff8-450e-b022-a87ff6bc301f_characterizeImage_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f12a8d9-cff8-450e-b022-a87ff6bc301f_characterizeImage_1039909_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:22 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956652e87e0b18d95d843e70f9ee7159 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 629f65719dfc664b4968ab4b7d15dd53 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8d4a57bf-5769-4c45-8a04-9d65ee32963e_calibrate_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8d4a57bf-5769-4c45-8a04-9d65ee32963e_calibrate_1039909_19.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/3b43458c-457b-46b5-8d51-c61947538c3c_characterizeImage_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/3b43458c-457b-46b5-8d51-c61947538c3c_characterizeImage_1039909_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:22 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec16ecbb73ec525ac5db56f2836abf23 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 03c09043b79ef0211b0af5b3886ec661 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bda7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/896136c9-e272-4e42-8aa3-8f0ce485551c_calibrate_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/896136c9-e272-4e42-8aa3-8f0ce485551c_calibrate_1039907_38.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7612296f-8653-4c3c-a4e5-9e1c8d636533_characterizeImage_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7612296f-8653-4c3c-a4e5-9e1c8d636533_characterizeImage_1039907_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:23 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e5a97577e651f0ee8f5ea1f6528919 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 8da54a12e67d8677c418bfdf5da21ffe parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f441c91d-3acb-4cb0-b048-8fee4fbb8e39_calibrate_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f441c91d-3acb-4cb0-b048-8fee4fbb8e39_calibrate_1039909_30.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6a746962-b052-4054-884a-e9e3fcdd9a4e_characterizeImage_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6a746962-b052-4054-884a-e9e3fcdd9a4e_characterizeImage_1039909_30.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d21b4f9d475175587a1c90601477bb0 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash b163c2f8fd189c966c044139b66be99d parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5063683f-d13f-401e-a5a6-a065ff1afc35_calibrate_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5063683f-d13f-401e-a5a6-a065ff1afc35_calibrate_1039909_39.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f3c3b1bd-e5cb-4139-97ee-5cc18cb79ad3_characterizeImage_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f3c3b1bd-e5cb-4139-97ee-5cc18cb79ad3_characterizeImage_1039909_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:23 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:24 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 823f8539211999af5e7372f416a2c34f with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 49a1e4a20ba033b2b7f8c4765164d8ff parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9e721341-14b4-49b4-8445-32bf3b616166_calibrate_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9e721341-14b4-49b4-8445-32bf3b616166_calibrate_1039908_35.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7c5b7240-f2f9-4ecd-923a-4f8a32711fad_characterizeImage_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7c5b7240-f2f9-4ecd-923a-4f8a32711fad_characterizeImage_1039908_35.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2f47368dece0447bb5e32b91f95f79c with result from task 37 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:24 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 06c25a8b9b4b476f3ee6a729946df1ec parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/eda0504b-7a67-4997-b157-4e8752caddf9_calibrate_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/eda0504b-7a67-4997-b157-4e8752caddf9_calibrate_1039907_8.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/27eb8fac-7e4f-4ddc-94ef-6f69127531e7_characterizeImage_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/27eb8fac-7e4f-4ddc-94ef-6f69127531e7_characterizeImage_1039907_8.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f651a0a20532ee75d81a74513942daf with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash eeedb3168cf06b364abdead27477f994 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/099eb41e-705a-489a-9974-bc2e0fd38fa3_calibrate_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/099eb41e-705a-489a-9974-bc2e0fd38fa3_calibrate_1040013_6.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ff276af2-b134-4f52-89eb-ef8a10b757cd_characterizeImage_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ff276af2-b134-4f52-89eb-ef8a10b757cd_characterizeImage_1040013_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:25 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:25 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175c615926aa84c3e6383171b2b465c5 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 61f86ee3d5903451f3ed3fb5a87177b4 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/39739fb8-7aea-4a94-b905-d4cb8235c5b4_calibrate_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/39739fb8-7aea-4a94-b905-d4cb8235c5b4_calibrate_1040013_17.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/53f84eb9-7f88-4db2-84ce-d72396d88951_characterizeImage_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/53f84eb9-7f88-4db2-84ce-d72396d88951_characterizeImage_1040013_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:26 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:26 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bfb14fdf253c42471dbb5e004e012ec with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash dd22509240fd14bd7d8ffd19cc60da77 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a211025b-2abd-400e-91f7-47bcd97420ca_calibrate_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a211025b-2abd-400e-91f7-47bcd97420ca_calibrate_1039908_11.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/bc313123-d9a0-4150-a1da-95a7c4a0e60a_characterizeImage_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/bc313123-d9a0-4150-a1da-95a7c4a0e60a_characterizeImage_1039908_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:27 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e3601104613f6f7ec7c20b7e5b7a26 with result from task 49 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:27 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash f83447d16fa0d399f16bb8577ce85217 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/743e4f0c-1347-4577-bdbc-19e64f5c70b1_calibrate_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/743e4f0c-1347-4577-bdbc-19e64f5c70b1_calibrate_1039908_5.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/62b90bfe-e53a-45da-be53-95c63f153941_characterizeImage_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/62b90bfe-e53a-45da-be53-95c63f153941_characterizeImage_1039908_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:28 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:28 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49b41c5a724c006822054417ae7f3eb1 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash dc45de50e521c1e1fbbabca2a7d15a60 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/5bbd54bf-49bd-4d53-abde-171538de79a8_calibrate_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/5bbd54bf-49bd-4d53-abde-171538de79a8_calibrate_1039908_12.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/49a22813-172d-4c1d-8f9c-17493c4dd0fa_characterizeImage_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/49a22813-172d-4c1d-8f9c-17493c4dd0fa_characterizeImage_1039908_12.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d101393efa07eabfdd42429c686c1b07 with result from task 7 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:29 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash f6bf69eef1b0ee55891640c72d3bf299 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8148860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/fe916077-a8de-486b-ad32-603380fdfc08_calibrate_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/fe916077-a8de-486b-ad32-603380fdfc08_calibrate_1039909_56.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/75f89dc5-71e7-4a49-b06d-bd0c90239c36_characterizeImage_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/75f89dc5-71e7-4a49-b06d-bd0c90239c36_characterizeImage_1039909_56.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04d38860daedadf4544f924ac3068539 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash dec9f59b8db01ccaffed27dad9bc9551 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8122c7a0-4840-4160-aacd-06205b9afdec_calibrate_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8122c7a0-4840-4160-aacd-06205b9afdec_calibrate_1039908_47.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cffe65fd-be45-4b5c-ab75-58bc7f436dc7_characterizeImage_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cffe65fd-be45-4b5c-ab75-58bc7f436dc7_characterizeImage_1039908_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:29 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:30 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6792e621c9e86a5ee0d0a43cdca5a616 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 0eec0e48a4374508561c8fcdfbf77c78 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/00cc2a20-38c9-40c3-bb2f-41d45f6b0640_calibrate_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/00cc2a20-38c9-40c3-bb2f-41d45f6b0640_calibrate_1039909_24.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8e85db91-e8c8-4d48-abff-ffdcd816f09b_characterizeImage_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8e85db91-e8c8-4d48-abff-ffdcd816f09b_characterizeImage_1039909_24.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3014242bd585bc34936a7aabc3360f60 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 4939c385784ba3cf0baab6ca394bd97c parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8c492599-81bc-4cae-8757-cc88e89e5913_calibrate_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8c492599-81bc-4cae-8757-cc88e89e5913_calibrate_1039909_47.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d85c9485-a913-4d12-88fa-5f9b25ab4eb2_characterizeImage_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d85c9485-a913-4d12-88fa-5f9b25ab4eb2_characterizeImage_1039909_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:30 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33ec52ff750481e1dbfc29414647fada with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7262d89aebe56f8e394089f1ca8517be parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/86baaf97-ad96-436b-b44a-a539ad1617c2_calibrate_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/86baaf97-ad96-436b-b44a-a539ad1617c2_calibrate_1040013_12.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cb380f92-d945-428b-9cba-9adfdfe83bc3_characterizeImage_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cb380f92-d945-428b-9cba-9adfdfe83bc3_characterizeImage_1040013_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:31 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 285ecb9337ee3ac60b83083f301595a4 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash ff21cecb0bccc841eb7af4cd7a78f5cf parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6783a328-897d-4676-9753-2124a6c20c58_calibrate_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6783a328-897d-4676-9753-2124a6c20c58_calibrate_1039909_62.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a2ed190f-bb30-4edd-a36b-4abd0cd80069_characterizeImage_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a2ed190f-bb30-4edd-a36b-4abd0cd80069_characterizeImage_1039909_62.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ebe4a314eccb8178a8c01940f544dac with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 21a3e4b64984415c7859e1fce6472fd5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8b82ed8f-bf6f-4716-a813-49e7effa8dce_calibrate_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8b82ed8f-bf6f-4716-a813-49e7effa8dce_calibrate_1039909_29.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5a2aff30-b933-4e36-be1a-0f6345a24448_characterizeImage_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5a2aff30-b933-4e36-be1a-0f6345a24448_characterizeImage_1039909_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:31 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:32 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:32 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5d1e2b2f25a768bf65f6cd61d810e8 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash c9eb05e3c02050d9204396ffa0f1c177 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/6fb71e44-0ca8-4f40-a2d1-089c8d28dfe1_calibrate_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/6fb71e44-0ca8-4f40-a2d1-089c8d28dfe1_calibrate_1040013_8.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5f9c7a18-af6b-4f2b-b39c-00eea0169aae_characterizeImage_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5f9c7a18-af6b-4f2b-b39c-00eea0169aae_characterizeImage_1040013_8.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f31575f990e81bee3191b30813e93d5 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash c2cfabeb7d566caec1888e18803554eb parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dbc0be10-4dd4-4916-b0e0-e1336c842f8d_calibrate_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dbc0be10-4dd4-4916-b0e0-e1336c842f8d_calibrate_1039909_46.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c02a4e53-2997-402d-b51a-eedfa312a17d_characterizeImage_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c02a4e53-2997-402d-b51a-eedfa312a17d_characterizeImage_1039909_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:33 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:33 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ba8a72d2c867dc26cbe0889abf559e with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash c5c9e270830d3e7ec114c883f7091316 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2f774426-df53-4f76-9fd0-d5d9ffc94a4a_calibrate_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2f774426-df53-4f76-9fd0-d5d9ffc94a4a_calibrate_1039909_52.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c0f50cf3-9d01-4c53-a480-723a0a7e0f11_characterizeImage_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c0f50cf3-9d01-4c53-a480-723a0a7e0f11_characterizeImage_1039909_52.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa76dcfc155b505cfbc23148c972dee with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash f8033931dd7a24ecc781c70c6950fb54 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d0f31115-5122-4016-8e7e-9f50777184a9_calibrate_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d0f31115-5122-4016-8e7e-9f50777184a9_calibrate_1039908_25.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/8a8a6287-4ecb-40b0-978d-52f787a92285_characterizeImage_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/8a8a6287-4ecb-40b0-978d-52f787a92285_characterizeImage_1039908_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:34 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aaa5d790a05b8d91b25bbba778c2651 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 2e5ee406c16f153dd75f6f40f37ed3e7 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/38f4f8f1-3de1-4cb2-84c8-859a2cd890f7_calibrate_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/38f4f8f1-3de1-4cb2-84c8-859a2cd890f7_calibrate_1039907_35.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0c69e746-52c2-441d-9106-9d7f107611a3_characterizeImage_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0c69e746-52c2-441d-9106-9d7f107611a3_characterizeImage_1039907_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:34 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:35 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffc8f03c9203fb36e251e3fa487945ec with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash d32ced14ceccef0c9d6564abde81ac5b parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81680e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/88d08a47-e48e-4772-af9c-483e56087f90_calibrate_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/88d08a47-e48e-4772-af9c-483e56087f90_calibrate_1039908_27.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d92d25d3-04f3-499c-8a38-751d141c1cc7_characterizeImage_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d92d25d3-04f3-499c-8a38-751d141c1cc7_characterizeImage_1039908_27.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85c62d9e2d8f0ba2d43783177e2138e0 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 97f67339663d93cbe3603b8018fe5dba parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b86a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e2ddada-bea7-417e-b765-c53b7f00e631_calibrate_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e2ddada-bea7-417e-b765-c53b7f00e631_calibrate_1040013_56.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4fe56190-09fe-43cf-b474-68644b2a49af_characterizeImage_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/4fe56190-09fe-43cf-b474-68644b2a49af_characterizeImage_1040013_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:35 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:36 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5605d3fa0bd2a1065c433aee790080be with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash e5d08e1be334259402b6630817c0bdac parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a24e61da-9361-4bc9-989b-cd47e13f08c8_calibrate_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a24e61da-9361-4bc9-989b-cd47e13f08c8_calibrate_1039909_12.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f28596d-678f-4fdc-81be-f4eeba760a2a_characterizeImage_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8f28596d-678f-4fdc-81be-f4eeba760a2a_characterizeImage_1039909_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:36 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:36 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:37 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:37 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:38 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf6fac9baed9034ccf52375c9e881b7f with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 73c902705a959df64ac6f53dfd6d66b2 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b7ed9573-68c9-4a56-971d-588d3501b876_calibrate_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b7ed9573-68c9-4a56-971d-588d3501b876_calibrate_1039908_29.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dafa59bf-1839-41d0-9e3f-35acde20db4d_characterizeImage_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/dafa59bf-1839-41d0-9e3f-35acde20db4d_characterizeImage_1039908_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:38 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e261e300468228411cd2b5b2914327d4 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7165f83c3e0ad3fa7236bb43272fe852 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2ca0c8d6-cc8b-4229-a9c4-5dd38c843383_calibrate_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2ca0c8d6-cc8b-4229-a9c4-5dd38c843383_calibrate_1039909_37.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1d0ace9b-7da4-4859-9ecf-7b686a79c433_characterizeImage_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1d0ace9b-7da4-4859-9ecf-7b686a79c433_characterizeImage_1039909_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:38 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d2a7c27f7afc9a1e7a3183b0ea6257 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash b4c69c20591454f00799604eeed836e1 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8a917eeb-eed1-4ed5-b977-25818aa6e192_calibrate_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8a917eeb-eed1-4ed5-b977-25818aa6e192_calibrate_1039909_15.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/bb61843b-d702-410e-b06d-bb87c25e7cbe_characterizeImage_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/bb61843b-d702-410e-b06d-bb87c25e7cbe_characterizeImage_1039909_15.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86b223ff3acaea5292c171ee89e1b047 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 1c92479d18c64c50c665164cbdf4af6d parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c7e52392-69ca-407c-b6fb-8e11b9161727_calibrate_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c7e52392-69ca-407c-b6fb-8e11b9161727_calibrate_1039907_31.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/09eb3e31-7811-46f9-a465-48ab58d584f1_characterizeImage_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/09eb3e31-7811-46f9-a465-48ab58d584f1_characterizeImage_1039907_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:39 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7c3d4280a3e2c55aaab439b3dbb3bc6 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash f81734600cc01e47b5fffeb844b2ae95 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e5f7afc0-d049-42ef-a515-15517542b689_calibrate_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e5f7afc0-d049-42ef-a515-15517542b689_calibrate_1039908_41.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a4377e4b-349e-413e-a925-914c6302b2fe_characterizeImage_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a4377e4b-349e-413e-a925-914c6302b2fe_characterizeImage_1039908_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:40 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:40 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7cf0afe329d39487790d3f95aeca5ed with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash ccf27632ccbe02807bb6b08788c121e5 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cac158b3-3cc6-4019-9d20-91cb8730ff7a_calibrate_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cac158b3-3cc6-4019-9d20-91cb8730ff7a_calibrate_1039909_1.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7890fba9-872a-4ef5-8205-80f08ad4dc02_characterizeImage_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7890fba9-872a-4ef5-8205-80f08ad4dc02_characterizeImage_1039909_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:40 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:41 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e591e1191163124b522ab240ed1905 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 167d6b88296de76d9986bef5fb8efdaf parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/666ae569-43e9-443f-8ea7-4ee377b7c746_calibrate_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/666ae569-43e9-443f-8ea7-4ee377b7c746_calibrate_1039909_18.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/12088e65-adda-4f83-8dea-c9583bb6407b_characterizeImage_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/12088e65-adda-4f83-8dea-c9583bb6407b_characterizeImage_1039909_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:41 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bb08bc177d7df2334b3d3f523c7b72c with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 8383efaeae44b2bd2ecd24bc28361e23 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8168a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/68c1060c-d59d-4bb8-a7a2-d626fcc97197_calibrate_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/68c1060c-d59d-4bb8-a7a2-d626fcc97197_calibrate_1039909_27.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/67c0b68a-b39d-4de9-b695-485cd688da08_characterizeImage_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/67c0b68a-b39d-4de9-b695-485cd688da08_characterizeImage_1039909_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:42 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:42 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:42 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73b7a8ec388539a8525a3e54233f2fcc with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash d37a95920bdcc4288466313acd5b3538 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dfb5e21a-872f-437f-a330-bff833f74e08_calibrate_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dfb5e21a-872f-437f-a330-bff833f74e08_calibrate_1039909_16.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a349c54d-0ede-4678-b935-5f0a402c8082_characterizeImage_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a349c54d-0ede-4678-b935-5f0a402c8082_characterizeImage_1039909_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:43 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc6c798f4b701cda0e03e4b961ed5949 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 17fd65abd06d4bd673c5aa03c3598dae parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/85aaec65-eff7-475e-b756-7f6e5fc04c6f_calibrate_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/85aaec65-eff7-475e-b756-7f6e5fc04c6f_calibrate_1039908_40.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a277e9fa-d6ff-4466-abd8-159712c51eec_characterizeImage_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/a277e9fa-d6ff-4466-abd8-159712c51eec_characterizeImage_1039908_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:43 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:44 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e084e3eb97e2308f4ce1d76fc7d2a47 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 26a432797ec90c065b3f523779e4cf16 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/03627040-5583-4a9e-b96b-c4eda1cf78e7_calibrate_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/03627040-5583-4a9e-b96b-c4eda1cf78e7_calibrate_1040013_34.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/11178b2d-0e10-447d-b40b-b042a37dc27f_characterizeImage_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/11178b2d-0e10-447d-b40b-b042a37dc27f_characterizeImage_1040013_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:44 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5ac92d14f513bb520bde52d0f3f737 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash dd51b5f210597af9edec9047a6707642 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8169080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4d63c130-73ab-4720-9ee6-80dfce0ce712_calibrate_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4d63c130-73ab-4720-9ee6-80dfce0ce712_calibrate_1039908_56.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/54c101cf-02d5-473f-88b2-38b5e572e9f6_characterizeImage_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/54c101cf-02d5-473f-88b2-38b5e572e9f6_characterizeImage_1039908_56.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad03519895df4f1dc3bf601cd2900cb8 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 0db0d1f28889efb36393f11941809082 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b50499e9-439f-40f5-a94b-7681f9b390d5_calibrate_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b50499e9-439f-40f5-a94b-7681f9b390d5_calibrate_1039908_49.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cbcf0f3e-18eb-48a8-b165-45431fe379c2_characterizeImage_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/cbcf0f3e-18eb-48a8-b165-45431fe379c2_characterizeImage_1039908_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:45 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f52073b8badf37326c366da3fb86a11 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash d2db4ad72b6ee0d9350db8cb9070fede parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f814ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f8e4eb86-3d02-4c2b-b254-e21aab7cb90c_calibrate_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f8e4eb86-3d02-4c2b-b254-e21aab7cb90c_calibrate_1040013_11.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f73066d8-5307-472d-8d38-f66e332362d5_characterizeImage_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f73066d8-5307-472d-8d38-f66e332362d5_characterizeImage_1040013_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:45 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:45 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:46 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c3434d8fc12dfc276d29f1446b5fee with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 1feed64ee237b809e45d505f3c703c92 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/07c67ede-1cdc-4c7d-9ecf-109f52702422_calibrate_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/07c67ede-1cdc-4c7d-9ecf-109f52702422_calibrate_1039909_58.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/58f9b8c2-280d-4d20-bf2c-f95823230ab2_characterizeImage_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/58f9b8c2-280d-4d20-bf2c-f95823230ab2_characterizeImage_1039909_58.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c2ba69a45fa0442ce848276e1cc94fe with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 740299c6f90f09d4dd9601e3351bc543 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/22062234-394a-4120-8f2a-d93e2d1d51d2_calibrate_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/22062234-394a-4120-8f2a-d93e2d1d51d2_calibrate_1039909_45.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/03d9f05a-1237-40e8-866e-c220a53a175d_characterizeImage_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/03d9f05a-1237-40e8-866e-c220a53a175d_characterizeImage_1039909_45.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e74a81e918bef8de7f41d213fdfc867 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash c11b0defe82c21c660ef58352b62f172 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8b8d7a39-25ae-4ef9-bace-1f4c0c66d4d3_calibrate_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8b8d7a39-25ae-4ef9-bace-1f4c0c66d4d3_calibrate_1039908_38.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0a392c9c-762b-4a4b-b72a-aeba432f1154_characterizeImage_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/0a392c9c-762b-4a4b-b72a-aeba432f1154_characterizeImage_1039908_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7f2ea0f12813485ac47c5892fdfd55a with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 76313b6ba76797290ad5eadd86d44ffb parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/126964bf-7520-49a6-aae1-9a063f71d1ef_calibrate_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/126964bf-7520-49a6-aae1-9a063f71d1ef_calibrate_1039909_9.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8fca7a7a-bfd3-4335-9246-402d0ac66d74_characterizeImage_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/8fca7a7a-bfd3-4335-9246-402d0ac66d74_characterizeImage_1039909_9.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5cc063462a946ab9e75c32ce817e24 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 071e4a13d71ec6861f462bb40ecf4718 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81885e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/73117f83-7aa2-4856-9e15-c037f71ab8b0_calibrate_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/73117f83-7aa2-4856-9e15-c037f71ab8b0_calibrate_1040013_21.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0379cb32-101d-4862-b825-797092f1e110_characterizeImage_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0379cb32-101d-4862-b825-797092f1e110_characterizeImage_1040013_21.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f1040a9cb169a35e26cb83b27a28337 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash b18189e3bd1385f5e8c9a03168f56804 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/05547afc-4275-4668-9fd7-5d6dfcb81d30_calibrate_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/05547afc-4275-4668-9fd7-5d6dfcb81d30_calibrate_1039907_26.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/a523ad3e-5ea3-40ed-be18-3c8664d1dc5b_characterizeImage_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/a523ad3e-5ea3-40ed-be18-3c8664d1dc5b_characterizeImage_1039907_26.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b534662597802c61c8a054d5a13569b8 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 56870bcdf9572daf3f85c649d51fcc9d parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f3c9eeee-24b8-468a-903e-1ab463032d02_calibrate_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f3c9eeee-24b8-468a-903e-1ab463032d02_calibrate_1039908_13.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/669ee119-7448-490b-8ba9-0b14783eb255_characterizeImage_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/669ee119-7448-490b-8ba9-0b14783eb255_characterizeImage_1039908_13.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67bc41663fbba0fb51e4c923e633bea8 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash c08e615ce4334c9d2b57d66002b73284 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c518a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/97ab4760-ab68-470e-b7b7-112f265bc1b6_calibrate_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/97ab4760-ab68-470e-b7b7-112f265bc1b6_calibrate_1039908_22.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6cc5a5a7-e7f2-4a12-8520-55869e35caad_characterizeImage_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6cc5a5a7-e7f2-4a12-8520-55869e35caad_characterizeImage_1039908_22.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2da8d7c728bfc322fced537fd4c5b2cf with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 99ca5516e9dd22e8e01cc4fc35acda65 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/21bbbd1a-807d-465f-8686-f6db6b9cc7ab_calibrate_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/21bbbd1a-807d-465f-8686-f6db6b9cc7ab_calibrate_1039909_28.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e9e61741-86e5-48e8-a943-dd56e149fd3b_characterizeImage_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e9e61741-86e5-48e8-a943-dd56e149fd3b_characterizeImage_1039909_28.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134f1020310f61607d5e295b0100d5d8 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash fbfa7b36296ba531580fa4ff2edaa627 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/90134e5a-71a0-408e-90cb-480f77050dc8_calibrate_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/90134e5a-71a0-408e-90cb-480f77050dc8_calibrate_1039907_44.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b0f0f6ce-72b4-4546-930b-bde349fc640b_characterizeImage_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b0f0f6ce-72b4-4546-930b-bde349fc640b_characterizeImage_1039907_44.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7810b5be86b860e689355ae783e7a59a with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 4199cf89d04f77a644e1aea98f960684 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dc8d32ca-4145-491f-a32d-0f6b21bf50ba_calibrate_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dc8d32ca-4145-491f-a32d-0f6b21bf50ba_calibrate_1039908_9.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/20068437-9529-4885-9350-c2f2e2a2d6ad_characterizeImage_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/20068437-9529-4885-9350-c2f2e2a2d6ad_characterizeImage_1039908_9.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84aa13b4c24a7dfc511e706686f705fa with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 9554e7a144b3084d35d4a969784af54c parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/51630cb9-da03-41d6-9760-0e29b8bb6b5d_calibrate_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/51630cb9-da03-41d6-9760-0e29b8bb6b5d_calibrate_1040013_54.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1754517e-482b-4a0e-8809-abeb9158392e_characterizeImage_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1754517e-482b-4a0e-8809-abeb9158392e_characterizeImage_1040013_54.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa3b55ae4f8602ed041e5e3ca349b71 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash a91a07b10b845a140d3b1fd176a8baa0 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/82680853-d121-47d6-810b-6c4aa2d76ca5_calibrate_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/82680853-d121-47d6-810b-6c4aa2d76ca5_calibrate_1040013_24.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f15f4332-64a8-4d02-82ac-03f98ef484c6_characterizeImage_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/f15f4332-64a8-4d02-82ac-03f98ef484c6_characterizeImage_1040013_24.stderr 2024-03-18 05:23:52 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:23:52 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169e28f1318d1b2a4dac9265a533b290 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 6ca487d1ae7283eba4f8459fc07e7c12 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/15abf8ea-2e50-4ec6-89d7-c134410de001_calibrate_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/15abf8ea-2e50-4ec6-89d7-c134410de001_calibrate_1040013_51.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b4925220-5f06-4c34-8103-051b0f5c8e9e_characterizeImage_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b4925220-5f06-4c34-8103-051b0f5c8e9e_characterizeImage_1040013_51.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc8373de0aec686d63adfb53701b1ba with result from task 436 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:52 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash a01a86da87fab94f9bfd5c04f6a84288 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/fcd10c40-e65f-46c4-8975-f4558349ccf4_calibrate_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/fcd10c40-e65f-46c4-8975-f4558349ccf4_calibrate_1039907_15.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5545a2b9-de7e-4e42-93b2-9b9dc8fae9a6_characterizeImage_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5545a2b9-de7e-4e42-93b2-9b9dc8fae9a6_characterizeImage_1039907_15.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf0259d21c2a4eab50bcc91a0e736433 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 618ab32f736b18e4ac9b71f1d6936f41 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dba19710-2767-4f13-8808-3f916927e44f_calibrate_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dba19710-2767-4f13-8808-3f916927e44f_calibrate_1039909_32.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f8940dfd-73ad-482e-9e1c-9eeed2700920_characterizeImage_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f8940dfd-73ad-482e-9e1c-9eeed2700920_characterizeImage_1039909_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:52 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:52 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648d6d1ecfde161e2cb99959c591187a with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash b7881124e11fe86a686b45666c7670e7 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9cd884fe-ff90-4a8f-ad09-8e0bc3d654e8_calibrate_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9cd884fe-ff90-4a8f-ad09-8e0bc3d654e8_calibrate_1040013_16.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/85c9f9b6-c657-4339-8499-9f31ce20e544_characterizeImage_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/85c9f9b6-c657-4339-8499-9f31ce20e544_characterizeImage_1040013_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:53 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 504ef47ce77eb3868f4bd6b4c98bd17f with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 4b9373b832ca20d1da45ccfa1ca4eb6b parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/41ab65f0-587e-477c-94a9-8292217c4169_calibrate_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/41ab65f0-587e-477c-94a9-8292217c4169_calibrate_1039907_40.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/136492fd-b853-40fd-a029-28dfbfc5718c_characterizeImage_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/136492fd-b853-40fd-a029-28dfbfc5718c_characterizeImage_1039907_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:53 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:54 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:54 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:54 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca36a1ecafc03c39577124341f9bc89a with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash e73fa1c8d3c0bd227e66de629747ae72 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b3d32285-2a01-417d-9f69-3aea0a32429f_calibrate_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b3d32285-2a01-417d-9f69-3aea0a32429f_calibrate_1039909_50.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/91becfbe-28ba-48d9-a02d-507b840f5ed7_characterizeImage_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/91becfbe-28ba-48d9-a02d-507b840f5ed7_characterizeImage_1039909_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:55 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:55 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:56 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c211b57cf2982329d3a549c9f6d6a0bb with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash fe11c4ccc2af570a617fafd8e3d04136 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c080e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/3d250521-2ba8-4c25-ad1a-57692af79979_calibrate_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/3d250521-2ba8-4c25-ad1a-57692af79979_calibrate_1040013_22.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cf9e7f65-07ca-4c6d-a553-844e6f083cc0_characterizeImage_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/cf9e7f65-07ca-4c6d-a553-844e6f083cc0_characterizeImage_1040013_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:56 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:57 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:57 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:57 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:58 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:58 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 399ca78267da9ed7848f6d1b523ff6fa with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 4ebd8b80e2c3e3d3467ae3438f8f1a13 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f81696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0f0ae67-3d9a-4efc-9fc1-30241c46b71c_calibrate_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0f0ae67-3d9a-4efc-9fc1-30241c46b71c_calibrate_1040013_46.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a89830e6-b08d-4499-bf5d-90d929d588ea_characterizeImage_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a89830e6-b08d-4499-bf5d-90d929d588ea_characterizeImage_1040013_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:58 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b17f8efe4721a51dc949bffdaac5d5da with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash acbf42717b1655678615171e7864cf47 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/491b0e5e-adf0-4aa3-8444-f457c379cac5_calibrate_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/491b0e5e-adf0-4aa3-8444-f457c379cac5_calibrate_1039907_3.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7b8dd8b4-7bc0-4174-ba8b-bb726c5c6464_characterizeImage_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/7b8dd8b4-7bc0-4174-ba8b-bb726c5c6464_characterizeImage_1039907_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:59 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17f43c997fc2c4186f0b2797f49d876d with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash af23cfb8e0309e3a388c1d9fb7cc507b parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c537e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/202761d5-5230-42f4-b396-b455bbd4e7ee_calibrate_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/202761d5-5230-42f4-b396-b455bbd4e7ee_calibrate_1039908_44.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/19db0644-a744-4db3-8a3b-d1ad91f566e3_characterizeImage_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/19db0644-a744-4db3-8a3b-d1ad91f566e3_characterizeImage_1039908_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:59 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 039f15cafbc0cd53f5d05b587acd6f8e with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash fcd5d80a1ad9ff558cc8fbacb92474a4 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d233ac97-bac1-4103-9a34-20fa44b2ad9d_calibrate_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d233ac97-bac1-4103-9a34-20fa44b2ad9d_calibrate_1039907_57.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f3dea3a0-d55f-41ac-820d-7b6f969a104d_characterizeImage_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/f3dea3a0-d55f-41ac-820d-7b6f969a104d_characterizeImage_1039907_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:23:59 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:00 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:00 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67c50966fba2768094a351c0c9d017c5 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash b2108594064fc9db884a6dd08dfd3885 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0815ce54-17bf-480d-bac6-28129d2c295e_calibrate_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0815ce54-17bf-480d-bac6-28129d2c295e_calibrate_1040013_40.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b6027595-ae34-40e8-a940-1fe6492d65be_characterizeImage_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/b6027595-ae34-40e8-a940-1fe6492d65be_characterizeImage_1040013_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:01 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:01 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac478422cd81bbcafbda2d4d10d6854 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash a582d5e7f41bcc83f1e1db3a74883fb8 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7a6c39a3-5d68-4fed-b466-c6b9dc8b32c3_calibrate_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7a6c39a3-5d68-4fed-b466-c6b9dc8b32c3_calibrate_1039907_50.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/8b2f16e2-eee2-465e-bfd3-de0a6f177003_characterizeImage_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/8b2f16e2-eee2-465e-bfd3-de0a6f177003_characterizeImage_1039907_50.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005b0235072d8148aeddeddee769784b with result from task 427 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:02 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 5e5b98c87db389d25d39588904945f6b parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b927927d-ea35-4ac4-baba-0b5ec477f09c_calibrate_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b927927d-ea35-4ac4-baba-0b5ec477f09c_calibrate_1039908_52.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/4f4cf4e5-0a1a-4ce5-b941-697e6dd6df7c_characterizeImage_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/4f4cf4e5-0a1a-4ce5-b941-697e6dd6df7c_characterizeImage_1039908_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:02 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:02 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b8face214aaf06c539c6eadeded7df4 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash a73a7e8acb703cc403bdacd8caa0ff9b parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e7c00cef-00e2-4686-8ac6-e6c7d1732e40_calibrate_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e7c00cef-00e2-4686-8ac6-e6c7d1732e40_calibrate_1039908_39.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1035e7c8-dc13-45a8-bbc1-a7e2616791b3_characterizeImage_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1035e7c8-dc13-45a8-bbc1-a7e2616791b3_characterizeImage_1039908_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:03 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c1f5887ad7fc7d209c90b224e3c7ee0 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash baab200a71902f9d448950caae17bee4 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2e5544f4-c4b8-4abb-93d2-d176dd7176c3_calibrate_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2e5544f4-c4b8-4abb-93d2-d176dd7176c3_calibrate_1039908_42.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7961127e-9cbf-47f5-ac9e-c15c924c21e9_characterizeImage_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7961127e-9cbf-47f5-ac9e-c15c924c21e9_characterizeImage_1039908_42.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9df015751f1b438e42ba7bb02e240caa with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash a47de037823dd689e45a4068bbd150c3 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/808d3170-1a73-4ac8-9387-3bdcf05331a6_calibrate_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/808d3170-1a73-4ac8-9387-3bdcf05331a6_calibrate_1039907_11.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/77e8a355-bb3b-46a5-80de-cf453d59ab6a_characterizeImage_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/77e8a355-bb3b-46a5-80de-cf453d59ab6a_characterizeImage_1039907_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:03 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4429c1a6e9ef6a9e247fac4cefa5d812 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 5108507f13bbd8f0838c5524716e9ab8 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3a4c1eeb-1099-4bb0-aa15-12d986a2fa93_calibrate_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3a4c1eeb-1099-4bb0-aa15-12d986a2fa93_calibrate_1039908_50.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/78e639c5-3eda-47f2-a9d1-29a81e6056ff_characterizeImage_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/78e639c5-3eda-47f2-a9d1-29a81e6056ff_characterizeImage_1039908_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:04 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:04 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:04 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9909a22cbee9b5823ea5c33cb46df544 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 4347055ef00e5dbdac3b202c9ae29b5b parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c81e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d5334784-b661-4f49-989a-2e61dac03c99_calibrate_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d5334784-b661-4f49-989a-2e61dac03c99_calibrate_1039909_3.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d0605ef0-adef-4d7c-a134-0be1d4a85c37_characterizeImage_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/d0605ef0-adef-4d7c-a134-0be1d4a85c37_characterizeImage_1039909_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:05 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 796638ea19555088582cdd0449866003 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash cebde6fd0f46a5741d64d7b0c1a7b530 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c807c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0ad27ecf-385f-4211-983a-32f66d82f01d_calibrate_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0ad27ecf-385f-4211-983a-32f66d82f01d_calibrate_1039909_51.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3a4fef5-e0fe-46ab-9531-77fd24d80b1e_characterizeImage_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a3a4fef5-e0fe-46ab-9531-77fd24d80b1e_characterizeImage_1039909_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:05 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 779dad62d8e9f086ab81de9f01542443 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 45c08d47daf76e7e1e2c06d6b690f334 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2a914ba5-d692-45fa-941f-51062b43e07c_calibrate_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2a914ba5-d692-45fa-941f-51062b43e07c_calibrate_1040013_41.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3115e165-a899-436d-8705-f2c3847f8a44_characterizeImage_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3115e165-a899-436d-8705-f2c3847f8a44_characterizeImage_1040013_41.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5152b6b362bd3cc08499d3b6d6945da2 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 01998b416066a429b5dd7eb94c977461 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2fe9b871-3e46-457a-beed-94a25b1c71ce_calibrate_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2fe9b871-3e46-457a-beed-94a25b1c71ce_calibrate_1040013_50.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a44ec457-a316-45e7-9586-2ed091f59147_characterizeImage_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a44ec457-a316-45e7-9586-2ed091f59147_characterizeImage_1040013_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 1 failed parsl.dataflow.dflow ERROR: Task 166 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 194db5cd5efd7c9cb5f3c6c310c6d8f8 with result from task 166 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 167 try 0 failed parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7609000b-4192-4fe5-bbc7-516435327398_calibrate_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7609000b-4192-4fe5-bbc7-516435327398_calibrate_1039907_2.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/202a0312-e35d-4647-86e3-6c2bea765983_characterizeImage_1039907_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:06 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04a27200a8423c8119a72236cd55ee13 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 154231b9630d337fad121183e4ad97d5 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/33404284-9fe8-4334-9f91-a89b4b1030a4_calibrate_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/33404284-9fe8-4334-9f91-a89b4b1030a4_calibrate_1040013_3.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ed7506e9-4b3b-4482-9b2d-e970976c2cf2_characterizeImage_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ed7506e9-4b3b-4482-9b2d-e970976c2cf2_characterizeImage_1040013_3.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc724aee0b3ced73940a4f4d118eddf6 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash be05cad7b28649eba602e5087ddc5fd2 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f08919b8-671f-4c6c-92ae-b9f67fa6be12_calibrate_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f08919b8-671f-4c6c-92ae-b9f67fa6be12_calibrate_1039908_28.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/32fe4d51-1b2a-441c-8555-c591d2770698_characterizeImage_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/32fe4d51-1b2a-441c-8555-c591d2770698_characterizeImage_1039908_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:06 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:06 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95cec5877e3d6515a56bd235702edfc9 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 7154d53060f4b849500ca6c168951074 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c093a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d91f7ab6-5870-41c5-a1de-982f7dbb2bba_calibrate_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d91f7ab6-5870-41c5-a1de-982f7dbb2bba_calibrate_1039908_24.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9737aab4-d385-4748-9c34-40eb24876526_characterizeImage_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9737aab4-d385-4748-9c34-40eb24876526_characterizeImage_1039908_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:07 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:07 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f153c14f51fd4ceeb3cb5a9f329a1a8 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash a8f48de504d9beb0119d7b770e6b9e09 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f818a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0276b19-dc45-4776-b93f-1dbf7e5e5ddf_calibrate_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0276b19-dc45-4776-b93f-1dbf7e5e5ddf_calibrate_1040013_25.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/38fe02b5-0c6b-4e68-8d32-73260afe162c_characterizeImage_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/38fe02b5-0c6b-4e68-8d32-73260afe162c_characterizeImage_1040013_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:08 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:08 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:09 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2516c31fece08c0a2f975ef8a12cdba5 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash dc29bbc1424f55a66b73eb8d20d094d2 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/72c08a90-4828-48ca-b554-ac47b4b14c39_calibrate_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/72c08a90-4828-48ca-b554-ac47b4b14c39_calibrate_1039909_35.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0f516198-cb5a-46eb-81af-1f11f3f596da_characterizeImage_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/0f516198-cb5a-46eb-81af-1f11f3f596da_characterizeImage_1039909_35.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 182aaa8762f8509d26d252fcab8ea7af with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash c647879cc73a13df20ab05d49878bc66 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c811c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c4c5975e-9b9f-4988-add0-7a0a10477d0a_calibrate_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c4c5975e-9b9f-4988-add0-7a0a10477d0a_calibrate_1039907_9.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bd12a380-450b-4338-97f2-e6285d54372c_characterizeImage_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bd12a380-450b-4338-97f2-e6285d54372c_characterizeImage_1039907_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:09 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ef74dd8460385a858270d1f1b37de9 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 7f39d256db81029ddda15580962c7097 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b87060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ba77b11e-e1fd-4884-9192-ceec1867afb8_calibrate_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ba77b11e-e1fd-4884-9192-ceec1867afb8_calibrate_1040013_38.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aa2d8a32-4d7a-4bf3-ad2b-11f8ee3b046c_characterizeImage_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aa2d8a32-4d7a-4bf3-ad2b-11f8ee3b046c_characterizeImage_1040013_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:09 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:10 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 454d3818f94a8aa0edbeca246b2bb98d with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash df3898a8696e330e8918597342e22bd5 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5ced387f-55cc-4054-a3bc-64dc25451625_calibrate_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5ced387f-55cc-4054-a3bc-64dc25451625_calibrate_1039909_57.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/cee0f2c9-c2b1-4911-8f8c-1b60ec6a6fb8_characterizeImage_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/cee0f2c9-c2b1-4911-8f8c-1b60ec6a6fb8_characterizeImage_1039909_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:10 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5246c9814a02f7e08ebb41a562fb9c7f with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash c405cbe0b92ef1da43c309b2dadc1db0 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d68a9427-bc5b-4d1f-a84c-a53cbd798113_calibrate_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d68a9427-bc5b-4d1f-a84c-a53cbd798113_calibrate_1039907_43.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87c68071-d84d-4e9b-8b4b-601014b9c657_characterizeImage_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87c68071-d84d-4e9b-8b4b-601014b9c657_characterizeImage_1039907_43.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 025636b908ed25ab3646e17c00c1e48b with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash f59a3b351509e79c9f62f96885136e51 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/51d82ac5-d4ef-4e8f-8459-0ea0b4e507af_calibrate_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/51d82ac5-d4ef-4e8f-8459-0ea0b4e507af_calibrate_1039909_41.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a8ff4a5a-7ec5-4ed6-a4e5-fb8ca0af81e0_characterizeImage_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/a8ff4a5a-7ec5-4ed6-a4e5-fb8ca0af81e0_characterizeImage_1039909_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:11 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:11 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f546d6373bca0b146e806dcba2f75078 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 05f2f4df4d0afa1d91b90407e15c1efe parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f32044e4-c0ff-4092-ac5b-2419eb7c86ef_calibrate_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f32044e4-c0ff-4092-ac5b-2419eb7c86ef_calibrate_1039907_51.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/49b01d4c-2fa4-4d19-b576-8a9dca7e94f3_characterizeImage_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/49b01d4c-2fa4-4d19-b576-8a9dca7e94f3_characterizeImage_1039907_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:12 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39bc61e17dd31c49779dee3a3e2d522c with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 499fae0af570c0bf43803ae067a7466c parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b84360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f0dcfc1d-29a5-4402-b014-1d9195f63f7e_calibrate_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f0dcfc1d-29a5-4402-b014-1d9195f63f7e_calibrate_1040013_4.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/990258c9-e84c-45c9-8186-fcb2f3c99996_characterizeImage_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/990258c9-e84c-45c9-8186-fcb2f3c99996_characterizeImage_1040013_4.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb721cfa3aa30e2b840466b9804f357 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 379d3985fc0e94b3a7f70339baf9832e parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6baf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4f79d61f-bb4e-4c61-ba10-4c8081cd5866_calibrate_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4f79d61f-bb4e-4c61-ba10-4c8081cd5866_calibrate_1040013_44.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/06fc3381-a027-4a62-9226-65e5c8ddb1c3_characterizeImage_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/06fc3381-a027-4a62-9226-65e5c8ddb1c3_characterizeImage_1040013_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:12 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15b264d09e9d165d4762ec1dfb954437 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash ac7bfc6e309a4502684d085bc23fa060 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/485ea2dd-ab41-49d7-a897-62a26fd9184e_calibrate_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/485ea2dd-ab41-49d7-a897-62a26fd9184e_calibrate_1040013_18.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/48e2e3ab-d285-4f01-9192-56de12169287_characterizeImage_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/48e2e3ab-d285-4f01-9192-56de12169287_characterizeImage_1040013_18.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d474f9f636a48ea6211c1dc889d7c76 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 3a2959a51d55b7773100cbcb99a8fcae parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b4bbe2e4-81a6-431b-a64f-6541853b0940_calibrate_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b4bbe2e4-81a6-431b-a64f-6541853b0940_calibrate_1039909_42.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/fcc8618b-b0a9-4426-8e40-770248bf11d8_characterizeImage_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/fcc8618b-b0a9-4426-8e40-770248bf11d8_characterizeImage_1039909_42.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ebea1dbfface0338b70965132c2087e with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 04ded2a93f5412636c3ebeadfdda45ea parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9e958cd-7ad6-446a-bdc4-769664f19ee4_calibrate_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9e958cd-7ad6-446a-bdc4-769664f19ee4_calibrate_1039909_17.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f78dd498-0816-44f9-a562-6949294b99eb_characterizeImage_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f78dd498-0816-44f9-a562-6949294b99eb_characterizeImage_1039909_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:13 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0cc338c87da79fc80aca6a00231d731 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash a20b0bcf76f293b4fb34500e78b7187f parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bd91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9f2f92b7-565e-4aff-8e1d-fdbd023ba8d9_calibrate_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9f2f92b7-565e-4aff-8e1d-fdbd023ba8d9_calibrate_1039907_29.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5165940e-8e8c-408b-92cf-51ee0c0f34d9_characterizeImage_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/5165940e-8e8c-408b-92cf-51ee0c0f34d9_characterizeImage_1039907_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:13 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 209fd5f465b141f90c711083d36b08b0 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 252f38fd22acb2dde3c667378cd2a332 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b85da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ff697bc2-7f16-4bf7-acc9-11b9e10816a1_calibrate_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ff697bc2-7f16-4bf7-acc9-11b9e10816a1_calibrate_1040013_57.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dfb32041-7c35-4c18-b34f-cf2cceceb59b_characterizeImage_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dfb32041-7c35-4c18-b34f-cf2cceceb59b_characterizeImage_1040013_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:13 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:14 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11717654afe2a80b1c33fcbdcd1878eb with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 234cda71e4012baff51f853f4bcbb55e parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/063992a3-e304-4c8b-8833-30849060e9cb_calibrate_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/063992a3-e304-4c8b-8833-30849060e9cb_calibrate_1039907_37.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0fad67ea-0916-42cb-b89c-67e5d62746b1_characterizeImage_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0fad67ea-0916-42cb-b89c-67e5d62746b1_characterizeImage_1039907_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:14 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:15 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:15 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e95bc2834e5d92ab7676d8aeb98c90f1 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash da82c421a4107cc38b5ef1585fc958ba parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b32153dc-ac18-4aa2-bbf6-c5749c4d0031_calibrate_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b32153dc-ac18-4aa2-bbf6-c5749c4d0031_calibrate_1039908_57.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/941293df-fe04-4026-84a7-b18ed1a745cd_characterizeImage_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/941293df-fe04-4026-84a7-b18ed1a745cd_characterizeImage_1039908_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:16 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88ab133b212e805b293a69a2df1cadf6 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash fee519dee78c3503091fa751f396c427 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0d217f02-a3a6-4f2e-add3-d825f5a3fae7_calibrate_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0d217f02-a3a6-4f2e-add3-d825f5a3fae7_calibrate_1040013_31.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1b768abd-9190-49a8-bf73-b3354fc7ab8c_characterizeImage_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1b768abd-9190-49a8-bf73-b3354fc7ab8c_characterizeImage_1040013_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:16 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 247 parsl.dataflow.strategy DEBUG: Executor multi has 247 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 247 parsl.dataflow.strategy DEBUG: Executor multi has 247 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80a72e7d30eb1c0fd946e7893ea287b5 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 30b18881948272cc5d8c02c866c294c5 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bacfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3c346e06-9cbb-4ea7-83a0-f14e42a6ca8c_calibrate_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3c346e06-9cbb-4ea7-83a0-f14e42a6ca8c_calibrate_1039907_60.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/92000987-b047-403b-a2a9-7efd76598748_characterizeImage_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/92000987-b047-403b-a2a9-7efd76598748_characterizeImage_1039907_60.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbd5f25a9dc1ee8231d66cc5eba9e5e with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash f58ef47f6e60f754d42f87c7183e6e9d parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/6372c030-5c17-4750-9add-ff42cadd7c49_calibrate_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/6372c030-5c17-4750-9add-ff42cadd7c49_calibrate_1039907_14.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6245c8d7-c425-4fcb-a585-0134d102e070_characterizeImage_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6245c8d7-c425-4fcb-a585-0134d102e070_characterizeImage_1039907_14.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e6a242c022007639d24ffd6345b081 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 1a4367ffbf8e8eabd49f841c35e1405c parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/de35ef51-b4d3-4e25-8a80-df5fe68ed562_calibrate_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/de35ef51-b4d3-4e25-8a80-df5fe68ed562_calibrate_1039909_8.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7bc4eb3c-ecf9-4592-9126-83e3b62d5023_characterizeImage_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/7bc4eb3c-ecf9-4592-9126-83e3b62d5023_characterizeImage_1039909_8.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1de31c2ae1de6476e28a06e0c88b30b7 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 6f7ea18103e399550f8baa6e44e2516d parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c099e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/ebfef480-995b-4f56-9c46-c364ebd9f6b6_calibrate_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/ebfef480-995b-4f56-9c46-c364ebd9f6b6_calibrate_1039907_47.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0489728b-a24a-41af-a02a-acd22888cf83_characterizeImage_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/0489728b-a24a-41af-a02a-acd22888cf83_characterizeImage_1039907_47.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7e82dc35339981b61c2d550757bf4da with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 49c07df91dbe299f7b40fa9eac549553 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6badc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cb0a127d-9fe3-4406-947d-c188ab31b1e0_calibrate_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cb0a127d-9fe3-4406-947d-c188ab31b1e0_calibrate_1039909_26.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2de57b8b-39c1-45e2-9d11-e3cff2c65bce_characterizeImage_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2de57b8b-39c1-45e2-9d11-e3cff2c65bce_characterizeImage_1039909_26.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4fe3b38229afe37162eafece727fad2 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 61417c797e9606b77c0c9a16bb5b3f52 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f816a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/153114a1-71db-42a8-94d6-654eb06b8d72_calibrate_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/153114a1-71db-42a8-94d6-654eb06b8d72_calibrate_1040013_58.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5879ce49-6958-4d35-bfc3-5abcbf5d0744_characterizeImage_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5879ce49-6958-4d35-bfc3-5abcbf5d0744_characterizeImage_1040013_58.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9e22ed1897882b430e30b59d0ffd118 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 7fe8f69a5b34d58d8db6f57c4f9de488 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bdba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fb606ff5-66e5-4f7e-be99-f0a95d11fc06_calibrate_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fb606ff5-66e5-4f7e-be99-f0a95d11fc06_calibrate_1040013_29.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9a24d5a1-627c-4120-973e-9948c7731295_characterizeImage_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9a24d5a1-627c-4120-973e-9948c7731295_characterizeImage_1040013_29.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8efce52be14163ad9eba64d702ba3134 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 24f263a38a5e8a423e7d1db07f3088bb parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bafb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/65853974-0320-4552-9996-a96c33119abb_calibrate_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/65853974-0320-4552-9996-a96c33119abb_calibrate_1039907_10.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1dd176e2-0a79-42df-8753-c95ee86d1061_characterizeImage_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1dd176e2-0a79-42df-8753-c95ee86d1061_characterizeImage_1039907_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 211 try 1 failed parsl.dataflow.dflow ERROR: Task 211 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 28ab71a4e222f92ac8fb949aee29f933 with result from task 211 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 212 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 212 try 0 failed parsl.dataflow.dflow INFO: Task 212 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/c4846d96-900f-47c7-9199-477f0230e09a_calibrate_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/c4846d96-900f-47c7-9199-477f0230e09a_calibrate_1040013_2.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a1aba71d-4987-43de-addb-24222582255a_characterizeImage_1040013_2.stderr 2024-03-18 05:24:22 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d452741045466cfba3b2c8015f5ed6c with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash b0b506ab1d983c00d052abfe0f358612 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/03a43b33-eec0-4598-9618-83870cb4b8c6_calibrate_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/03a43b33-eec0-4598-9618-83870cb4b8c6_calibrate_1039908_37.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/074ae1fb-e217-4801-ab64-5fd0e7fe6ee1_characterizeImage_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/074ae1fb-e217-4801-ab64-5fd0e7fe6ee1_characterizeImage_1039908_37.stderr 2024-03-18 05:24:22 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 352ba78f6d9206551cef49c3c007cf22 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 8f4822245c892a6e69e9e573838a76cf parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c80ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4154cdd0-6eeb-47c1-ba58-83f46eeb6146_calibrate_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4154cdd0-6eeb-47c1-ba58-83f46eeb6146_calibrate_1039908_45.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/e0897aac-5146-4bb7-821e-a409d14452aa_characterizeImage_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/e0897aac-5146-4bb7-821e-a409d14452aa_characterizeImage_1039908_45.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80c31c2f483a4bef4eda47690fb4fb98 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash ecf3ba8c766673f3fab48aaa2a1851fa parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/daa8b94b-3951-4934-a18f-1b14cba22352_calibrate_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/daa8b94b-3951-4934-a18f-1b14cba22352_calibrate_1040013_23.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/feddbed3-b9c3-4f79-8271-9476325a7b7c_characterizeImage_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/feddbed3-b9c3-4f79-8271-9476325a7b7c_characterizeImage_1040013_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:22 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:23 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:23 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7064eed8598db84179af6f377cb20fcd with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 13694b0518ef9f55898e4d625bc9b055 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c009b0b6-0710-4079-a54a-5d760b83912c_calibrate_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c009b0b6-0710-4079-a54a-5d760b83912c_calibrate_1039909_22.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5b9f4e7a-8888-46d2-ada7-83e2f11b9b71_characterizeImage_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5b9f4e7a-8888-46d2-ada7-83e2f11b9b71_characterizeImage_1039909_22.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685398addbe8fdb31fab313077a04d4a with result from task 274 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:24 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 2e149be91941dc49101b72f10feca158 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6bae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/171cb9d0-f9a7-4fa2-9ee5-cbc5664aaf34_calibrate_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/171cb9d0-f9a7-4fa2-9ee5-cbc5664aaf34_calibrate_1039908_58.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/b163ed18-b6d7-46cf-ba1f-dd744b0ce132_characterizeImage_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/b163ed18-b6d7-46cf-ba1f-dd744b0ce132_characterizeImage_1039908_58.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a2f78a03c96271bd5d0f52fe1efebd with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 3fac201175ca911dd72a663384f077ac parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9a62e1ab-e713-4b91-b3d1-a4ef8bf4fd94_calibrate_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9a62e1ab-e713-4b91-b3d1-a4ef8bf4fd94_calibrate_1039907_36.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6872601b-2e3e-4982-9b6c-abe4ec184ce1_characterizeImage_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/6872601b-2e3e-4982-9b6c-abe4ec184ce1_characterizeImage_1039907_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:24 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:25 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:25 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:25 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987ec4319ffb04c1e481e65bdc93df4f with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 55b06b562d02fc413903fe287db1c18c parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c09080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/abf3e5b5-5759-4579-b507-800dbbed7547_calibrate_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/abf3e5b5-5759-4579-b507-800dbbed7547_calibrate_1040013_13.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0913cea5-1c18-4d7c-b34e-516734e69ed1_characterizeImage_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/0913cea5-1c18-4d7c-b34e-516734e69ed1_characterizeImage_1040013_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 331 try 1 failed parsl.dataflow.dflow ERROR: Task 331 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cc87dcef0be0080c7329819e7855e780 with result from task 331 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 332 try 0 failed parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f205c9af-b0fb-4785-b665-97b3bf68b856_calibrate_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f205c9af-b0fb-4785-b665-97b3bf68b856_calibrate_1039909_2.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e3e1ff27-67d5-4694-9ede-bd56e180e7f5_characterizeImage_1039909_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:26 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:26 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b63f361fcbfca3e5842e772e5e83e93 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 72be6b8aca4e99b606c57cfe819dd437 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c51760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/18421d18-a8dd-4ff7-abce-3a91dea7c1ed_calibrate_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/18421d18-a8dd-4ff7-abce-3a91dea7c1ed_calibrate_1040013_14.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/2e8838ae-e055-4186-82ea-8efef59c6440_characterizeImage_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/2e8838ae-e055-4186-82ea-8efef59c6440_characterizeImage_1040013_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:27 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:27 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:27 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:28 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:28 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10c3a44a4d24db6c278651563b50835c with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 4f4f29957cc465de1d8ec5e410ff2587 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46c54c81-afe5-4d4f-899b-d56b2b8bdd1f_calibrate_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46c54c81-afe5-4d4f-899b-d56b2b8bdd1f_calibrate_1039908_10.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7a23695a-5b01-49fc-a505-ffa692667e4f_characterizeImage_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7a23695a-5b01-49fc-a505-ffa692667e4f_characterizeImage_1039908_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:29 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e054a9c8300adf63c77bc36bb70621c with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 82568e021314a6471718f0cb691e8792 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f6081528-0391-4662-88ee-ddd516557e8a_calibrate_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f6081528-0391-4662-88ee-ddd516557e8a_calibrate_1039907_39.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/48cc0076-213e-47ac-8d7e-3f10280af503_characterizeImage_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/48cc0076-213e-47ac-8d7e-3f10280af503_characterizeImage_1039907_39.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6755e157d5d875c6d7d8b96696687f2d with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 62953e95001673fbfb5b7d7c7dccb1c9 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d314df02-75d4-4b9b-adf0-eb2f046284bb_calibrate_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d314df02-75d4-4b9b-adf0-eb2f046284bb_calibrate_1039907_53.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ab0b6418-acc7-40d7-a137-7fd2cf3ecb6a_characterizeImage_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/ab0b6418-acc7-40d7-a137-7fd2cf3ecb6a_characterizeImage_1039907_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:29 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cdfa07104e8985c127b4f603a540eb7 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 675a95ef02a0db1d82bf209ce79ec2da parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6addb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bf2b3a9d-aff8-4d2d-b386-f2f3c5955511_calibrate_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bf2b3a9d-aff8-4d2d-b386-f2f3c5955511_calibrate_1039908_7.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fd11efac-2d72-4f9f-ac2c-65853fb4a439_characterizeImage_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/fd11efac-2d72-4f9f-ac2c-65853fb4a439_characterizeImage_1039908_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:30 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:30 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9acf9516974063589efad2504b143eb1 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 641203232fed82915d1241c4d1d29fac parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f2fe57b1-82fa-4f1d-ad42-ee15ceb63f40_calibrate_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f2fe57b1-82fa-4f1d-ad42-ee15ceb63f40_calibrate_1039909_23.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6ba6ceb9-d7e3-4015-8525-ef3126079739_characterizeImage_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/6ba6ceb9-d7e3-4015-8525-ef3126079739_characterizeImage_1039909_23.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 220772dff681aa159f6887468d8baa83 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash b5bcb1736f6423fdb9a9737dc0945129 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b100e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bd9147a-99de-4aab-92e3-4f52154f35a4_calibrate_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bd9147a-99de-4aab-92e3-4f52154f35a4_calibrate_1039909_6.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/9f614824-edda-4e3b-bae8-33a4098778ad_characterizeImage_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/9f614824-edda-4e3b-bae8-33a4098778ad_characterizeImage_1039909_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:31 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:31 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:32 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78ce932db5b002cf3244611439faf2f6 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 6d8790ff5c098d062c45363a4800bd8b parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ae47bc07-150e-4ee8-909d-0530c0a01e01_calibrate_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ae47bc07-150e-4ee8-909d-0530c0a01e01_calibrate_1039909_25.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/dbfc2e8f-6281-48f5-9a11-08aeec3e02ab_characterizeImage_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/dbfc2e8f-6281-48f5-9a11-08aeec3e02ab_characterizeImage_1039909_25.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb9e237f47c9e45f3285b9c227c48a3 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash b9c721ec9d290c01914a894ad95b9f44 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/55ee9186-02d5-4617-a1a7-1b06548da7e7_calibrate_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/55ee9186-02d5-4617-a1a7-1b06548da7e7_calibrate_1039907_32.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/269e31e1-afb5-4f46-8377-037dd73ecb44_characterizeImage_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/269e31e1-afb5-4f46-8377-037dd73ecb44_characterizeImage_1039907_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:33 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ece3298e63800a1450f2d5bfbdd2ac92 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 116397f9e38f2b05a5190f33806808b8 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d03c70f4-4664-4082-9045-ce8d932f45b3_calibrate_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d03c70f4-4664-4082-9045-ce8d932f45b3_calibrate_1039909_38.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c8122163-0c1d-40e0-921a-093a1d30e0e0_characterizeImage_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/c8122163-0c1d-40e0-921a-093a1d30e0e0_characterizeImage_1039909_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:33 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1add067a091974e751120721bd8bba22 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 10166c8a8bede1da0324d9e5fc0d3534 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2c0e29c5-aa02-4b58-9582-9e200eb3ee77_calibrate_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2c0e29c5-aa02-4b58-9582-9e200eb3ee77_calibrate_1039907_19.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/760952bd-a64e-4a60-ba3d-5f7d92a4fe63_characterizeImage_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/760952bd-a64e-4a60-ba3d-5f7d92a4fe63_characterizeImage_1039907_19.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff7d6e8a00890d724fa031165d4d6e2e with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 005b5ac1da93bfcc2c65fa52630d6caa parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b4b47508-26b5-4be0-9288-42869aed04c1_calibrate_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b4b47508-26b5-4be0-9288-42869aed04c1_calibrate_1040013_9.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96e4b657-374d-4ee3-8302-ea89b4ca532d_characterizeImage_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96e4b657-374d-4ee3-8302-ea89b4ca532d_characterizeImage_1040013_9.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 1 failed parsl.dataflow.dflow ERROR: Task 154 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 342858d2be62085ffb45a8427efa2341 with result from task 154 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 155 try 0 failed parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/0f4df6c7-b627-4b89-98fd-714b388600a1_calibrate_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/0f4df6c7-b627-4b89-98fd-714b388600a1_calibrate_1039907_61.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/dc360aa7-bf75-4a30-a83f-62f900c5c110_characterizeImage_1039907_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:34 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e153a4b5df833bca9be18e2d10d33d3 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 310c6f37b2ea533708fee4ed67471b6c parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0d798f34-f2e7-452a-9946-3f10bf15903d_calibrate_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0d798f34-f2e7-452a-9946-3f10bf15903d_calibrate_1039909_53.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1250c0b7-b8a5-4f67-ad8c-b6d76e5611a0_characterizeImage_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/1250c0b7-b8a5-4f67-ad8c-b6d76e5611a0_characterizeImage_1039909_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:34 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5148bd3b95719bb07819cf18d75d4909 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 52f9b410a537049987dd8a0915bec513 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ed549143-0dea-4576-9c9d-64fcd383fde3_calibrate_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ed549143-0dea-4576-9c9d-64fcd383fde3_calibrate_1039909_48.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/abeb1280-b946-4371-ab5c-62e1ea84e077_characterizeImage_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/abeb1280-b946-4371-ab5c-62e1ea84e077_characterizeImage_1039909_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:34 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35de1f363418c514a50e3eeb9dc51299 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 29a31db3e6303607a1cc18fd4bd448db parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/83898f83-71f1-4350-adfe-aab9e31321d1_calibrate_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/83898f83-71f1-4350-adfe-aab9e31321d1_calibrate_1039907_20.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/9e84c40d-e755-4364-98fc-fdfaf25ca59b_characterizeImage_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/9e84c40d-e755-4364-98fc-fdfaf25ca59b_characterizeImage_1039907_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:35 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:35 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8002af7f51e506daf7ffc1acb41502e with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 3e696c0e4047435c0ad543d2ce7f8575 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c52340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2a5e0a77-0ac9-4148-8c53-a859288249f2_calibrate_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2a5e0a77-0ac9-4148-8c53-a859288249f2_calibrate_1039907_25.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/487cb2b4-07a2-46c2-adfb-077fe2e3c229_characterizeImage_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/487cb2b4-07a2-46c2-adfb-077fe2e3c229_characterizeImage_1039907_25.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c84c7d2a757283687682ffbceb0b147 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 9b9a0e31116e8108b1f3a5a66d95fbc3 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/99d81a2c-efe0-4972-bb53-d250b8e1e73b_calibrate_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/99d81a2c-efe0-4972-bb53-d250b8e1e73b_calibrate_1039907_46.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/79815dbc-d4e8-49ce-a10e-b69ff96f33de_characterizeImage_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/79815dbc-d4e8-49ce-a10e-b69ff96f33de_characterizeImage_1039907_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:36 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:37 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a89bb352c5706ccdbbf97340a768944 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 252fe5c91d12dd90142baa024521fa9f parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0208a3bf-bf62-4f66-8bbf-7ae131ed6aa8_calibrate_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0208a3bf-bf62-4f66-8bbf-7ae131ed6aa8_calibrate_1039908_46.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/86850980-a501-4264-ae52-ddc227b6794b_characterizeImage_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/86850980-a501-4264-ae52-ddc227b6794b_characterizeImage_1039908_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:37 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92134cee520e99918c9311f4ae4cb032 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash f69fefc0e46e3365298bb8aa9b0f4d49 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0e67f632-8b26-4de3-9d4f-6796754d3198_calibrate_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0e67f632-8b26-4de3-9d4f-6796754d3198_calibrate_1039909_5.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/025c9338-f33e-4802-ad5c-68d3934434ab_characterizeImage_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/025c9338-f33e-4802-ad5c-68d3934434ab_characterizeImage_1039909_5.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9865d269c4e11928c8c214a243aabd5 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash fdf654d12d83c8aba95650d4126c5f49 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/19892956-aacc-4de8-830b-530ae1cf47ba_calibrate_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/19892956-aacc-4de8-830b-530ae1cf47ba_calibrate_1039907_58.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bb86783b-5d81-42f9-aefd-6e8fb3b9c27f_characterizeImage_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/bb86783b-5d81-42f9-aefd-6e8fb3b9c27f_characterizeImage_1039907_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:38 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3c4e87d1389db9d572aa1969161657d with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 41448fc068e67aa4b47d2055d4e15cb7 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c82ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a0515838-4a34-4495-8ed0-06d3701955b8_calibrate_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a0515838-4a34-4495-8ed0-06d3701955b8_calibrate_1039907_56.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b798afc-c195-43cd-9978-93e63c6053c9_characterizeImage_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b798afc-c195-43cd-9978-93e63c6053c9_characterizeImage_1039907_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:39 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:39 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:39 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:40 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:40 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5c8f3736de7f35d8c1ba39962b986e8 with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash a5dc034a14a76403c4f8f6696d2d4744 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6ade980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cfae7756-588a-4567-8d25-30a737fd0778_calibrate_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cfae7756-588a-4567-8d25-30a737fd0778_calibrate_1039909_4.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/45b7ce20-7bab-41ba-b0df-56057794112c_characterizeImage_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/45b7ce20-7bab-41ba-b0df-56057794112c_characterizeImage_1039909_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:41 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac7419c51d49918d91c5d170eb3d4827 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 7c5d24e7026842c87fbb70c72b3b01d0 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c83a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f1afb7aa-5b5b-4fa1-83e7-7c6a1808a077_calibrate_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f1afb7aa-5b5b-4fa1-83e7-7c6a1808a077_calibrate_1040013_33.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9cfaa779-bc57-4714-9fdd-8d975d34cd7c_characterizeImage_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/9cfaa779-bc57-4714-9fdd-8d975d34cd7c_characterizeImage_1040013_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:41 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:42 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:42 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:42 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c23a70df0d08271b435119beb066dfb with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 66762972e7aa7771d09be295d0f42e16 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c08720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1700267-aba9-4e73-b261-729886aa4095_calibrate_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1700267-aba9-4e73-b261-729886aa4095_calibrate_1040013_55.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aff54db2-5392-4122-9d92-753e25bfadf7_characterizeImage_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aff54db2-5392-4122-9d92-753e25bfadf7_characterizeImage_1040013_55.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bbc0410563ffc2caabd9e8e2a7ff48d with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 6a55cdcbd1376be4a03bc9b4741b8bdf parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c827a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b238722e-d4ff-46fd-933f-01bdc86c2966_calibrate_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b238722e-d4ff-46fd-933f-01bdc86c2966_calibrate_1040013_43.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96d91338-ba85-4357-b343-a7dab8e2a5f3_characterizeImage_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/96d91338-ba85-4357-b343-a7dab8e2a5f3_characterizeImage_1040013_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:43 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96e64c5e2e6ec7c3e71ecb2e7228e08c with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 1ddd25d3fbac63287806597f73ebb8be parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b134c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a2f28eb7-fb19-4593-93b2-3cb5432b1ed1_calibrate_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a2f28eb7-fb19-4593-93b2-3cb5432b1ed1_calibrate_1039908_51.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/faab8337-f805-444d-b32d-30d874f6f827_characterizeImage_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/faab8337-f805-444d-b32d-30d874f6f827_characterizeImage_1039908_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:43 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:43 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:43 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:44 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:44 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad339f9ebd2d3d771fc0f97623de90c0 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash f48d05e5ec4f42ef131b8ea0742b2b50 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d11e184d-0e73-440e-8565-88d5696749a8_calibrate_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d11e184d-0e73-440e-8565-88d5696749a8_calibrate_1039908_19.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75b079e1-9a5b-47f9-8af9-88b87960d674_characterizeImage_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75b079e1-9a5b-47f9-8af9-88b87960d674_characterizeImage_1039908_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:44 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:44 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb9af39ecaef2ad7ffc5946880618daf with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash ca893badbf9b7d20f37d5b5eef00609b parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/85661529-1e7e-4329-abbf-dfe041de6048_calibrate_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/85661529-1e7e-4329-abbf-dfe041de6048_calibrate_1039907_30.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/db2ff676-9a65-43d7-a31a-dcd167b07dd1_characterizeImage_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/db2ff676-9a65-43d7-a31a-dcd167b07dd1_characterizeImage_1039907_30.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89142d4f7b908ae48f4385e5fe1badd2 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash c82b0d7c431827b66dd0481492169dd0 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b107c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bde7434-af15-42f9-8fe8-86ad043b1fed_calibrate_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bde7434-af15-42f9-8fe8-86ad043b1fed_calibrate_1039909_13.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/115632bb-2b61-41ea-ae5d-9bff461eb599_characterizeImage_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/115632bb-2b61-41ea-ae5d-9bff461eb599_characterizeImage_1039909_13.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca1f149ba597f2ebc5a3439e564a0e5 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 15a1bbdc081326647387cff412e571d8 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1f70dde3-c1d3-479b-9e03-94bdc0dcbe43_calibrate_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1f70dde3-c1d3-479b-9e03-94bdc0dcbe43_calibrate_1039907_55.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/12fcbe7b-21d2-4d06-8b09-4963c28627ce_characterizeImage_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/12fcbe7b-21d2-4d06-8b09-4963c28627ce_characterizeImage_1039907_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:44 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:45 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1002a05adbfd25f1d6a628bc591b7944 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash b70899772ca08597dbed844bd117e0d7 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c53880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4335f9a0-59c7-463a-b257-4a8ee5127608_calibrate_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4335f9a0-59c7-463a-b257-4a8ee5127608_calibrate_1040013_30.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a8027259-521d-42d8-abcb-dab758ad4ad5_characterizeImage_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/a8027259-521d-42d8-abcb-dab758ad4ad5_characterizeImage_1040013_30.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fceffe08ccdffc7592b30f4e85118278 with result from task 562 parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:45 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 8f0cc526aeb248070e702275970c3be1 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1a0098d9-be2f-4729-be0c-4574e82cca92_calibrate_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1a0098d9-be2f-4729-be0c-4574e82cca92_calibrate_1039908_59.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/df522365-2582-433a-b08b-67293fb8a007_characterizeImage_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/df522365-2582-433a-b08b-67293fb8a007_characterizeImage_1039908_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:45 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b108e14ad6ece34a19ac70840c6f5824 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/05f28d4f-058f-4bdb-801e-957ade3dbfc0_calibrate_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/05f28d4f-058f-4bdb-801e-957ade3dbfc0_calibrate_1039909_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 1 failed parsl.dataflow.dflow ERROR: Task 226 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca5729918013c6c6251f94dc60ba8c8 with result from task 226 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 227 try 0 failed parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8484797c-8d16-4b3b-b413-1e24f3428036_calibrate_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8484797c-8d16-4b3b-b413-1e24f3428036_calibrate_1039908_61.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1191110b-43b3-4001-b50a-d96b254aee54_characterizeImage_1039908_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:46 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f53d1c1234573da43113def60a0f639 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/454abbfc-4d78-4ca0-941e-eda3f9147a30_calibrate_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/454abbfc-4d78-4ca0-941e-eda3f9147a30_calibrate_1039907_45.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6df791aaf7aebadf9ec7ca324c528ad with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash bd596403af9aaf8f7cb2078b8c3e5f03 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/662a84f1-cee2-4c22-a753-2cdfcafd72fd_calibrate_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/662a84f1-cee2-4c22-a753-2cdfcafd72fd_calibrate_1039907_21.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/367a03dd-b9cc-46f1-b208-f25d202ea832_characterizeImage_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/367a03dd-b9cc-46f1-b208-f25d202ea832_characterizeImage_1039907_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:46 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c29e57b1d70a3ee8429e91d3d2d3408a with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 85df5046316f39e6082ea021893c19e9 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/60a533d0-22bd-45d4-acc7-99f0643e4a0b_calibrate_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/60a533d0-22bd-45d4-acc7-99f0643e4a0b_calibrate_1039907_42.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/19a95757-e342-4bdd-9cc6-7e466ef2c3c6_characterizeImage_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/19a95757-e342-4bdd-9cc6-7e466ef2c3c6_characterizeImage_1039907_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:47 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:47 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:47 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 191ee72eaf32ddced6f34c7add36e24c with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 7573fb4b4123d98d3c7f90579ea2c661 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dbad4805-54ae-4349-a6bb-c33a87ddc87c_calibrate_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dbad4805-54ae-4349-a6bb-c33a87ddc87c_calibrate_1039908_20.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/eeacf325-9215-4a2d-b754-674a23d3bde6_characterizeImage_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/eeacf325-9215-4a2d-b754-674a23d3bde6_characterizeImage_1039908_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 241 parsl.dataflow.strategy DEBUG: Executor multi has 241 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 241 parsl.dataflow.strategy DEBUG: Executor multi has 241 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 186f279f7d5016d9f6afd10d359d9b82 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 8f25502627b7a80d220165a28a08b4b4 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d615a00d-9784-4509-a5a4-a5eb2c950762_calibrate_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d615a00d-9784-4509-a5a4-a5eb2c950762_calibrate_1039907_23.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87500765-0bb8-49ea-b12c-2635fcd3d518_characterizeImage_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/87500765-0bb8-49ea-b12c-2635fcd3d518_characterizeImage_1039907_23.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fd12fd369efcc178a2b836b7db0f7d with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 18c0f54805969d52718ad437a98a4eaf parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/06dd8328-c5c5-4dd5-9c20-9bcd949592db_calibrate_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/06dd8328-c5c5-4dd5-9c20-9bcd949592db_calibrate_1039909_34.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/92ad5455-1fdc-48e1-a377-6412d0f61f8e_characterizeImage_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/92ad5455-1fdc-48e1-a377-6412d0f61f8e_characterizeImage_1039909_34.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3e278e86d97641d88f2bb2272a60599 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash d9723e63d517050bfd8c466e9fd88bb5 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3b52e5cc-1d60-481a-a406-7a112c104d4e_calibrate_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3b52e5cc-1d60-481a-a406-7a112c104d4e_calibrate_1039907_27.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1a420601-9236-4097-b7d9-306a2abad849_characterizeImage_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/1a420601-9236-4097-b7d9-306a2abad849_characterizeImage_1039907_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 526 try 1 failed parsl.dataflow.dflow ERROR: Task 526 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 31ee9e4bd65be9ffe300ba84161ab601 with result from task 526 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 527 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 527 try 0 failed parsl.dataflow.dflow INFO: Task 527 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2c8c7eeb-445b-4fb4-8f83-44cd39d3a938_calibrate_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2c8c7eeb-445b-4fb4-8f83-44cd39d3a938_calibrate_1039908_2.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/63ada1d8-8634-425e-9e1f-b9898607e11a_characterizeImage_1039908_2.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5add391ee4ced5d4b068fcddfc588a2 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f6924ff7-3fa6-430e-8e42-800ebfc80c78_calibrate_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f6924ff7-3fa6-430e-8e42-800ebfc80c78_calibrate_1039908_1.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fabd3c8c35e3868dde6d709df7c8a5fc with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 14072301b003caa4247018196bccf9da parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c50e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6cdfa6e8-7649-4e10-a9fd-8cb07648de13_calibrate_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6cdfa6e8-7649-4e10-a9fd-8cb07648de13_calibrate_1039909_59.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b15ed4a-2fa7-41bf-9c81-e88c0f1a1f2f_characterizeImage_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2b15ed4a-2fa7-41bf-9c81-e88c0f1a1f2f_characterizeImage_1039909_59.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03731cdceff55165c6de6850d342892 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/815d1031-3766-4803-9db4-16ee40a30e3a_calibrate_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/815d1031-3766-4803-9db4-16ee40a30e3a_calibrate_1039907_7.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c904c9e2c9ec39b532466609f0a24da with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 89ab8bfe689aa2586d769b1b60e32527 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0b2587cd-5c07-4464-8139-2218b3c51d7a_calibrate_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0b2587cd-5c07-4464-8139-2218b3c51d7a_calibrate_1039909_60.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f86450b6-e3ac-468f-9ddc-67d8bae23761_characterizeImage_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f86450b6-e3ac-468f-9ddc-67d8bae23761_characterizeImage_1039909_60.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d658ae630c71a6c71a2a3faaa82674be with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 9f63a6ce2f5a81a4c6480fa28d9c70ce parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2d55b173-f666-4825-bd42-a7559fc2ad90_calibrate_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2d55b173-f666-4825-bd42-a7559fc2ad90_calibrate_1039908_21.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6e8abb94-9e84-4830-81d7-cf11f0c9bec5_characterizeImage_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/6e8abb94-9e84-4830-81d7-cf11f0c9bec5_characterizeImage_1039908_21.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 233e2c3b211d089f92e315d3f4d173e1 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 5113c3406d8660604fa858d9e9199194 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/20ec4892-682f-4392-a3e6-3735b5147bba_calibrate_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/20ec4892-682f-4392-a3e6-3735b5147bba_calibrate_1039908_26.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7fe8bc3f-e087-4e5f-b28c-b8d626c9ef74_characterizeImage_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/7fe8bc3f-e087-4e5f-b28c-b8d626c9ef74_characterizeImage_1039908_26.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0200cc6f9f3ca884a0f92641b832ce98 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 607d0a65c1e03595254b017389db5dca parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c1a2eadb-ba25-4fb9-a48c-675341fc4407_calibrate_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c1a2eadb-ba25-4fb9-a48c-675341fc4407_calibrate_1039907_18.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b936b06-3318-485a-9131-7d511c5a48de_characterizeImage_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/4b936b06-3318-485a-9131-7d511c5a48de_characterizeImage_1039907_18.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15183786a8854cf7d602e9f82cb1df98 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash fd2eaa31975afcbde0e22e3209726e9e parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3d686b17-1995-4943-bebd-fa7f16e11e8f_calibrate_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3d686b17-1995-4943-bebd-fa7f16e11e8f_calibrate_1039907_17.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/76756253-b440-4733-ae4a-3f4142ad156f_characterizeImage_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/76756253-b440-4733-ae4a-3f4142ad156f_characterizeImage_1039907_17.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87019cae20985773522f0eae8800a0d3 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 52ed69d36f840b7e9514b362e007082e parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e67a5906-736a-4379-b9fe-b0794c01d2da_calibrate_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e67a5906-736a-4379-b9fe-b0794c01d2da_calibrate_1039908_62.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d9f8b0cb-4da7-44a9-b125-c13f5face8c0_characterizeImage_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/d9f8b0cb-4da7-44a9-b125-c13f5face8c0_characterizeImage_1039908_62.stderr 2024-03-18 05:24:53 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2c4de091f477e9b2d3a9e1d36d4df04 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/47ef03b3-91f2-47ac-99f7-de2e50e13517_calibrate_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/47ef03b3-91f2-47ac-99f7-de2e50e13517_calibrate_1039907_49.stderr 2024-03-18 05:24:53 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 689412462b68fbf8142861fcbcfb567f with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash a2c5f55831bea40b7227ee7ce3b2b623 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1fdff434-03bf-443e-b985-25a11dd65eff_calibrate_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1fdff434-03bf-443e-b985-25a11dd65eff_calibrate_1039908_43.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1d439f48-75f2-430e-99c9-d03add0a823a_characterizeImage_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/1d439f48-75f2-430e-99c9-d03add0a823a_characterizeImage_1039908_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:53 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc5f5d45ca342b836cbb7bdc49076b70 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2621eaa0-bade-4be9-a420-b752e672c6cb_calibrate_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2621eaa0-bade-4be9-a420-b752e672c6cb_calibrate_1039907_13.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8058374ab9939795d871bff215db25e with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9a90b2cc-99dc-4406-ade1-c48344bb6c20_calibrate_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9a90b2cc-99dc-4406-ade1-c48344bb6c20_calibrate_1039908_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:54 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996837020ded87f1f980073bd749b573 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3e29ae7b-f363-4334-bd3a-106e8fbed689_calibrate_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3e29ae7b-f363-4334-bd3a-106e8fbed689_calibrate_1039908_34.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d404ac56b7c09fcdfadf41a89e7378e with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 67f70877ee307150aaabaead19125703 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1e2198a0-6024-425c-b7c2-853c9be138d3_calibrate_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1e2198a0-6024-425c-b7c2-853c9be138d3_calibrate_1040013_52.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/bc160d69-7285-45c2-8ded-f264af90e571_characterizeImage_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/bc160d69-7285-45c2-8ded-f264af90e571_characterizeImage_1040013_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:55 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:55 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 585c6ba1ac926456faebcdc5ff20db0b with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 245469972203fcbc1711e082a8e0cdaa parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2e3c5d53-22d2-4147-9cac-c9b963cfe9d3_calibrate_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2e3c5d53-22d2-4147-9cac-c9b963cfe9d3_calibrate_1039907_6.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b394650d-db59-47d5-a877-191b956f0cfa_characterizeImage_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/b394650d-db59-47d5-a877-191b956f0cfa_characterizeImage_1039907_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:55 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67ebcbb3b6592ba1a62f17df201ca5ed with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 4e3256f9a1edafabfe337169d1270619 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/efe658b2-8fcc-4db6-8337-4f8ddc8b590b_calibrate_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/efe658b2-8fcc-4db6-8337-4f8ddc8b590b_calibrate_1039908_48.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/326aa9a4-c048-466f-932e-e04f7f04c3c1_characterizeImage_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/326aa9a4-c048-466f-932e-e04f7f04c3c1_characterizeImage_1039908_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:56 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3b91307ff2b365fa8c47429dbe48be5 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1d8b06be-5f81-4871-9572-9dda8febe43a_calibrate_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1d8b06be-5f81-4871-9572-9dda8febe43a_calibrate_1039907_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:56 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 909ed7e9e2cb9a58c03c0d4128de3b83 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/459cacff-4578-4191-a8c7-0bb147d566e6_calibrate_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/459cacff-4578-4191-a8c7-0bb147d566e6_calibrate_1039907_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:57 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:57 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:58 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24c528a7a2f1c312ef68fa0068dde0e1 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash d3c8c735bc426ea381e6d22a57502b56 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2d35a1fa-ccd1-4946-bcd6-887ce110cf79_calibrate_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2d35a1fa-ccd1-4946-bcd6-887ce110cf79_calibrate_1039907_5.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/fda35c94-42a1-4e36-a85f-d635692c7fb2_characterizeImage_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/fda35c94-42a1-4e36-a85f-d635692c7fb2_characterizeImage_1039907_5.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82ec651a99a780f292f7804b8bbd500c with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a903f7f1-c6ac-4de6-bfe9-62fdce0ac36f_calibrate_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a903f7f1-c6ac-4de6-bfe9-62fdce0ac36f_calibrate_1039909_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:58 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 681191ec70c7d0ae924a9c241899b07c with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 3db990a48c5167c8052198608f8b0106 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7ba7aa41-523e-46b6-a8b4-09a73cf5449e_calibrate_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7ba7aa41-523e-46b6-a8b4-09a73cf5449e_calibrate_1040013_20.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aedf62bb-cc1f-4f7b-9c29-fc166aaed26c_characterizeImage_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/aedf62bb-cc1f-4f7b-9c29-fc166aaed26c_characterizeImage_1040013_20.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 815a059dd5e13b4d3c7018d8f9891209 with result from task 17 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:58 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/df0e0507-d602-4693-b374-ac1e19832b4f_calibrate_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/df0e0507-d602-4693-b374-ac1e19832b4f_calibrate_1039909_43.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46941856b57bc29114d5ee56ea6faf37 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b576b967-0594-48c0-83f8-74131447f21d_calibrate_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b576b967-0594-48c0-83f8-74131447f21d_calibrate_1039908_31.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8654f406177b7138a07b8448b0727b1 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash d650f7410b6121456955a6be58da9875 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6aded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fabe2178-90fc-4977-99bb-95549f6cf75f_calibrate_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fabe2178-90fc-4977-99bb-95549f6cf75f_calibrate_1040013_49.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/7db49775-1cc2-4c66-80b3-0419a6838d7b_characterizeImage_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/7db49775-1cc2-4c66-80b3-0419a6838d7b_characterizeImage_1040013_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:59 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0099d01ecf9a961c3247c7397761de with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 782d85bde35a8f42cadaeb69f1c4aa6f parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c096c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e52c4cb4-6ce0-4bc4-81ea-e10b66136856_calibrate_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e52c4cb4-6ce0-4bc4-81ea-e10b66136856_calibrate_1040013_15.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1ee196e8-efe8-4d12-ad2c-ec791a6c5160_characterizeImage_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/1ee196e8-efe8-4d12-ad2c-ec791a6c5160_characterizeImage_1040013_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:24:59 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 687538a8cbdd1bba71fc18cf8ee8e4a6 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash f006f3cc282ded1c0c4b960030dec34b parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6badf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e2444c9e-c3ae-459a-b3d3-3cf7ec5f1338_calibrate_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e2444c9e-c3ae-459a-b3d3-3cf7ec5f1338_calibrate_1040013_60.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/54ae6c08-af1e-44e8-9377-ccd4bbeea691_characterizeImage_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/54ae6c08-af1e-44e8-9377-ccd4bbeea691_characterizeImage_1040013_60.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b28b14a4a9a326243e4db5bf6d48d0d with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/759a60f0-d689-4e41-a2ca-4c188a2cfd6a_calibrate_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/759a60f0-d689-4e41-a2ca-4c188a2cfd6a_calibrate_1039907_52.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 583beed3db74bbaa510a791b39890e89 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 1fdce20978fe045451a1a242a01555d7 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/14c8b6a8-61f1-4663-a409-682d359ecd7c_calibrate_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/14c8b6a8-61f1-4663-a409-682d359ecd7c_calibrate_1040013_37.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df1bc2a4-6a4b-4be9-8bda-d2a7a7da3728_characterizeImage_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df1bc2a4-6a4b-4be9-8bda-d2a7a7da3728_characterizeImage_1040013_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:00 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17caf2ddc8ae0c99f971f67c01e0d257 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/277bfc1e-3d6d-4fcd-b820-38be7d83cb2f_calibrate_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/277bfc1e-3d6d-4fcd-b820-38be7d83cb2f_calibrate_1039907_22.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c4a5c627e085c7a9cead4b97f4b7cd with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8d2fe28f-92ac-4e50-8272-2fe8c23390b3_calibrate_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8d2fe28f-92ac-4e50-8272-2fe8c23390b3_calibrate_1039908_8.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40fa87a5c9d2b2fc659187934d9881e6 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 4bee0e1756deae48ab6dab17f0349459 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/cb0458d9-ac2b-40c6-911d-df813e4abb4a_calibrate_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/cb0458d9-ac2b-40c6-911d-df813e4abb4a_calibrate_1039908_23.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/84c0d36f-37eb-40b6-b5c3-01a21cb6314c_characterizeImage_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/84c0d36f-37eb-40b6-b5c3-01a21cb6314c_characterizeImage_1039908_23.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a09a7a71a185d8749028c4174f81bba with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash c147af9f5baf8dd85de25ae6617f1ae7 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/71e4d1a6-cd20-4414-82fd-94d3975b1d30_calibrate_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/71e4d1a6-cd20-4414-82fd-94d3975b1d30_calibrate_1040013_27.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/761676c7-c97c-4a0a-9786-2527d841b10e_characterizeImage_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/761676c7-c97c-4a0a-9786-2527d841b10e_characterizeImage_1040013_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:01 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de953591d18744186f10618db944e390 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/cdd5d50a-a235-409c-b666-65c03913394a_calibrate_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/cdd5d50a-a235-409c-b666-65c03913394a_calibrate_1039907_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:01 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d728cd21c558bb3e6793af41eefa7cc with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash b7dfa482e0af876e3068ed34996eb977 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c814e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1a09c1c8-bb54-4c3b-8493-320cb12b39b9_calibrate_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1a09c1c8-bb54-4c3b-8493-320cb12b39b9_calibrate_1040013_59.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e0537053-33a8-4100-9933-0c2428dc60f9_characterizeImage_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e0537053-33a8-4100-9933-0c2428dc60f9_characterizeImage_1040013_59.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca507bc1c79a8a02edb156c4c2b8f4a with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 97bde07702e19d4521318c1e69395ed5 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/454aac97-d5be-4036-956b-8ee9618c9322_calibrate_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/454aac97-d5be-4036-956b-8ee9618c9322_calibrate_1040013_32.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/64dddf31-b639-40ac-9d33-926bdaf23ddb_characterizeImage_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/64dddf31-b639-40ac-9d33-926bdaf23ddb_characterizeImage_1040013_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:02 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99f23877dd42f2756a5bcd98e1fb3f39 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 9732366702c5a4f80f6f218130ed1e9c parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b12520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f833ac08-4b42-41c4-9e40-2c89d8a84d3c_calibrate_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f833ac08-4b42-41c4-9e40-2c89d8a84d3c_calibrate_1040013_10.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/c3e62604-6007-41ef-94d7-ada148e043c6_characterizeImage_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/c3e62604-6007-41ef-94d7-ada148e043c6_characterizeImage_1040013_10.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b77b6fc02e48a983a39a9fc7f641710e with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ff12cbf9-d445-41df-82c3-8cf4347e3c98_calibrate_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ff12cbf9-d445-41df-82c3-8cf4347e3c98_calibrate_1039909_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:03 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 858b0b3ee628f44220b0e73c7b30eef8 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4007fba1-4875-4322-80a2-5e21464eee1b_calibrate_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4007fba1-4875-4322-80a2-5e21464eee1b_calibrate_1039908_3.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47369ecce78dfca9bcfbfa5bb6e0e763 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 65ba1cbd222b40e2e1280f2f1eb0c116 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d2e967da-1d50-4494-baaf-ce2837757aaf_calibrate_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d2e967da-1d50-4494-baaf-ce2837757aaf_calibrate_1039907_24.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/902dd33e-97b6-48a0-9980-e5854a8110c1_characterizeImage_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/902dd33e-97b6-48a0-9980-e5854a8110c1_characterizeImage_1039907_24.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbcb6bf4810824e9348522204c4e1015 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4796a652-608e-48b7-9628-537e9dcd9851_calibrate_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4796a652-608e-48b7-9628-537e9dcd9851_calibrate_1039908_16.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2309b1547ac1e707211bb56c7a6350e3 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash e846cac7651566077c4924fee15f0a28 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/685f19d1-fabb-4b50-8287-c037e75975d3_calibrate_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/685f19d1-fabb-4b50-8287-c037e75975d3_calibrate_1039908_6.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9d7b1954-54b0-42d8-8d38-7ae5af529e52_characterizeImage_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/9d7b1954-54b0-42d8-8d38-7ae5af529e52_characterizeImage_1039908_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:03 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8cc19f2af7bc6a261ffc865de64c3a6 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash d5b8420765f3c6d2656599fc4c7e6c46 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a020c357-8fd1-4543-ad50-2100764d0c01_calibrate_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a020c357-8fd1-4543-ad50-2100764d0c01_calibrate_1040013_39.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df363eb6-bc8a-4004-bc5e-d59d207f679b_characterizeImage_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/df363eb6-bc8a-4004-bc5e-d59d207f679b_characterizeImage_1040013_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:04 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc7223cbc8619315cbce06af2d96f559 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 2ca5dafb4fecfb9825a0829bf8279892 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c691cf37-795b-4fef-9e11-e7d33d290da4_calibrate_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c691cf37-795b-4fef-9e11-e7d33d290da4_calibrate_1039909_55.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2e2af495-ee7c-4878-8a56-1335d818f567_characterizeImage_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/2e2af495-ee7c-4878-8a56-1335d818f567_characterizeImage_1039909_55.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5739a184c48ecf3bccc5d7a1d15b863 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d3cd410c-879e-4608-bb2f-a849857249e2_calibrate_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d3cd410c-879e-4608-bb2f-a849857249e2_calibrate_1039907_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:05 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 343 try 1 failed parsl.dataflow.dflow ERROR: Task 343 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 03d8ac9af6680731de874090147ac7ae with result from task 343 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 344 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 344 try 0 failed parsl.dataflow.dflow INFO: Task 344 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/57d551f1-4bc3-42f6-8c88-bd3b0bcf8487_calibrate_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/57d551f1-4bc3-42f6-8c88-bd3b0bcf8487_calibrate_1039909_61.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/e7e60303-e08c-402a-967b-5dec70af4603_characterizeImage_1039909_61.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bc073b12f6253fa5de7a0439659ef02 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/615363f0-4deb-4c52-9dbc-6fde8323daed_calibrate_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/615363f0-4deb-4c52-9dbc-6fde8323daed_calibrate_1039907_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:06 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0ff82641697ba6078fef508101be158 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash b190a45fe1d145139286a40ad0664ee3 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b118a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e089132e-29e8-449e-a629-e309152a92d7_calibrate_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e089132e-29e8-449e-a629-e309152a92d7_calibrate_1040013_5.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3b152bd1-6a5a-4dc6-8865-24cb00896474_characterizeImage_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/3b152bd1-6a5a-4dc6-8865-24cb00896474_characterizeImage_1040013_5.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab4fc1fd4e4118feb66d1b70fd6c63c with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash d70b2975784cd5e9519a8776e603af77 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0337acb0-fd1c-4736-ac9b-1d81d149f66c_calibrate_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0337acb0-fd1c-4736-ac9b-1d81d149f66c_calibrate_1039908_53.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/356c51d7-2c8a-42a1-ab67-7a7fc3b4755e_characterizeImage_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/356c51d7-2c8a-42a1-ab67-7a7fc3b4755e_characterizeImage_1039908_53.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610b08c98d9c022703559cf7a95bd30a with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/e1a287c5-b132-4e50-8a26-ddbe0c0e86b0_calibrate_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/e1a287c5-b132-4e50-8a26-ddbe0c0e86b0_calibrate_1039907_33.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8926e40875721033410401e4e291a579 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/aa17032e-1f38-4d23-bd34-e83ca57c9358_calibrate_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/aa17032e-1f38-4d23-bd34-e83ca57c9358_calibrate_1039909_36.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2ddb5f457de1a6c544b8dc25083fde with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 5e6e745fffee4588860a6356f04a8e26 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b137e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/67577e02-6184-483d-bbf4-431eb7eb35e4_calibrate_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/67577e02-6184-483d-bbf4-431eb7eb35e4_calibrate_1039908_60.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5220ad20-16ba-404f-893c-628cceb122de_characterizeImage_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5220ad20-16ba-404f-893c-628cceb122de_characterizeImage_1039908_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:07 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32f47c1d5a416002421be9db6ec07027 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash caf4e1cc265d10584136339ea31a839d parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d6fefe0b-b9af-44f8-9f99-d7d945641320_calibrate_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d6fefe0b-b9af-44f8-9f99-d7d945641320_calibrate_1039907_16.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/352e2781-dab4-4819-8945-acd3765ef681_characterizeImage_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/352e2781-dab4-4819-8945-acd3765ef681_characterizeImage_1039907_16.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d7eb5934572301e2b159919866fcbec with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/ee743ba7-9229-40f4-8449-d939952b3d89_calibrate_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/ee743ba7-9229-40f4-8449-d939952b3d89_calibrate_1039908_54.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c438f5bf8c7f4769f774194dcc4cc1d with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1c2e039d-6abf-4928-846a-26e7bd235089_calibrate_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1c2e039d-6abf-4928-846a-26e7bd235089_calibrate_1039908_15.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9b075c08e2f2ad8b237566def113470 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/280c77b8-5061-4df8-a9b9-f2afb6f16f51_calibrate_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/280c77b8-5061-4df8-a9b9-f2afb6f16f51_calibrate_1039909_33.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20ecdfc9f1c767607ea5b8232552bea with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/8dc37be1-ae18-42c6-8b8b-72da68f77d4f_calibrate_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/8dc37be1-ae18-42c6-8b8b-72da68f77d4f_calibrate_1039907_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:07 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:07 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:08 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:08 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1599b4c5a40d0717ab3ef63667ce248c with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/07f234c1-5bd9-4d7e-b536-3cb67212ce29_calibrate_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/07f234c1-5bd9-4d7e-b536-3cb67212ce29_calibrate_1039907_59.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83d6c89a9f8f50cd89064c94f1b2ee0 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bdc74dd7-ad31-4d1d-b27b-815b2c7fca32_calibrate_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bdc74dd7-ad31-4d1d-b27b-815b2c7fca32_calibrate_1039908_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:08 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7c7e9f2e92e80b74b67ea35bf159c8b with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/67ac85de-0628-46f0-ad79-25fa41e87892_calibrate_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/67ac85de-0628-46f0-ad79-25fa41e87892_calibrate_1039909_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:09 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:09 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:09 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb6b809d75bf61e4722229098585c1b9 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/e1ff9285-3ac5-458b-926e-dabd1ce69611_calibrate_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/e1ff9285-3ac5-458b-926e-dabd1ce69611_calibrate_1039909_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:09 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f83f23dea40359df9040e59ba35718 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash bddef693f3d07632765b47eaf36906a4 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1be981d7-6cad-4ca0-bd06-336f063ff7d1_calibrate_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1be981d7-6cad-4ca0-bd06-336f063ff7d1_calibrate_1039908_32.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/882ad7f4-b117-469a-8686-d1c7c2477c1a_characterizeImage_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/882ad7f4-b117-469a-8686-d1c7c2477c1a_characterizeImage_1039908_32.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b4bea1c273b441d70c046e7938fcc8e with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash cbcc5d55e2602bedf2866cb06ade0d9d parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1fe43ad2-e16f-48ef-ac51-5d979e8b92f3_calibrate_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1fe43ad2-e16f-48ef-ac51-5d979e8b92f3_calibrate_1039907_62.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/43e9f15b-9045-4ddb-a162-5c9be276d10b_characterizeImage_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039907/43e9f15b-9045-4ddb-a162-5c9be276d10b_characterizeImage_1039907_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:10 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac9b1ecd26467f56869188de39912d9c with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a722cad6-8868-473a-bb10-58917058f48a_calibrate_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a722cad6-8868-473a-bb10-58917058f48a_calibrate_1039907_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:10 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a265fa20182fdaff0a4852192fa5c696 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 4ed8762c2c351c812ba648ca356a4e7e parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e0eb1de-e0de-41a2-91c2-73a91020216f_calibrate_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e0eb1de-e0de-41a2-91c2-73a91020216f_calibrate_1040013_62.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/99206326-90dd-4260-9f00-09baffa1bdfc_characterizeImage_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/99206326-90dd-4260-9f00-09baffa1bdfc_characterizeImage_1040013_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:11 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:11 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:11 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eca8f3afe1111e6373f1e9e1d47ec12a with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash c48e4c46e9e7e208951f09f22eb053fa parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b104a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1bf3dfd0-a9a1-425e-a389-1f859ac7eca4_calibrate_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1bf3dfd0-a9a1-425e-a389-1f859ac7eca4_calibrate_1040013_36.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/fe723beb-c325-40f8-be55-647bb55c7341_characterizeImage_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/fe723beb-c325-40f8-be55-647bb55c7341_characterizeImage_1040013_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:12 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:12 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d9e3db6f98a2cd3b9977ab8a172fb15 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 2def4f85d1c7a5217da0339a85b61b06 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6add4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c6d4c696-bf76-4db5-96ac-0b30a4a8c276_calibrate_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c6d4c696-bf76-4db5-96ac-0b30a4a8c276_calibrate_1039908_30.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5145ce2c-2d6a-49b0-a253-06bb87404bc4_characterizeImage_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/5145ce2c-2d6a-49b0-a253-06bb87404bc4_characterizeImage_1039908_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:13 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:13 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:13 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:14 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c25a8b9b4b476f3ee6a729946df1ec with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/eda0504b-7a67-4997-b157-4e8752caddf9_calibrate_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/eda0504b-7a67-4997-b157-4e8752caddf9_calibrate_1039907_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:14 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccf3ca7102c5bea8dd020da1d28b1b5c with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2dd02ded-6fab-4faa-9581-8b14151ca26f_calibrate_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2dd02ded-6fab-4faa-9581-8b14151ca26f_calibrate_1039908_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:15 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d494180da2e23a5072df119240872e with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/4bf79b47-8f28-4c0c-aacc-c78e2edf01b6_calibrate_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/4bf79b47-8f28-4c0c-aacc-c78e2edf01b6_calibrate_1039907_1.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4311fe3de4aefde256141a9d9fde5f41 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c8ef4b28-bfce-440e-ba1c-a4185da32d9e_calibrate_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c8ef4b28-bfce-440e-ba1c-a4185da32d9e_calibrate_1039908_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:15 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03c09043b79ef0211b0af5b3886ec661 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/896136c9-e272-4e42-8aa3-8f0ce485551c_calibrate_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/896136c9-e272-4e42-8aa3-8f0ce485551c_calibrate_1039907_38.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69084ddb1eae5d07176d94ea519ed211 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash f9770e831e4cc8359f0cfcc0e17401fc parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/1ebc82b6-7660-4ac5-b42b-f928f8e09105_calibrate_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/1ebc82b6-7660-4ac5-b42b-f928f8e09105_calibrate_1039909_49.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5218424f-d511-4e72-af40-3f56a001038c_characterizeImage_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/5218424f-d511-4e72-af40-3f56a001038c_characterizeImage_1039909_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:15 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:16 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:16 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:16 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:16 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:16 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:17 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfb381560dd233ca4630b07867f7f0c0 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 0bfb843a4786bd26dc42faa048558c90 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7d175e88-f3e1-4d98-9423-1c8a9fa697da_calibrate_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7d175e88-f3e1-4d98-9423-1c8a9fa697da_calibrate_1039909_11.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/063acb30-78b2-4314-80ce-7f1b4005a592_characterizeImage_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/063acb30-78b2-4314-80ce-7f1b4005a592_characterizeImage_1039909_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:17 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa148237e7e8abe55fe0fc81b2d17c7 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/172d456e-2739-4933-baa3-60bed4d7eb61_calibrate_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/172d456e-2739-4933-baa3-60bed4d7eb61_calibrate_1039908_55.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c92479d18c64c50c665164cbdf4af6d with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c7e52392-69ca-407c-b6fb-8e11b9161727_calibrate_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c7e52392-69ca-407c-b6fb-8e11b9161727_calibrate_1039907_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:18 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e5ee406c16f153dd75f6f40f37ed3e7 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/38f4f8f1-3de1-4cb2-84c8-859a2cd890f7_calibrate_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/38f4f8f1-3de1-4cb2-84c8-859a2cd890f7_calibrate_1039907_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:18 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f83447d16fa0d399f16bb8577ce85217 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/743e4f0c-1347-4577-bdbc-19e64f5c70b1_calibrate_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/743e4f0c-1347-4577-bdbc-19e64f5c70b1_calibrate_1039908_5.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 057f8e748ddbc6c5cf58aab86017f48f with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash eb416e1dd82f5ca64a584dac0493b8f0 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6adc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/36611d25-2576-41c8-8321-e2f3dca6ea48_calibrate_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/36611d25-2576-41c8-8321-e2f3dca6ea48_calibrate_1040013_53.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/addf8558-fa68-4206-8e20-edebf64d431e_characterizeImage_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/addf8558-fa68-4206-8e20-edebf64d431e_characterizeImage_1040013_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:18 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:19 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:19 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3eaf651c1d1d23face4c31e70a1a917 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash dda0a3da51fc3f0404b79bffb61a50d2 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/42ac8433-4e76-4c64-aadb-763277a4850e_calibrate_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/42ac8433-4e76-4c64-aadb-763277a4850e_calibrate_1040013_26.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/33ed53c3-0fd1-4779-b50e-eff43f669a58_characterizeImage_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/33ed53c3-0fd1-4779-b50e-eff43f669a58_characterizeImage_1040013_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 199 parsl.dataflow.strategy DEBUG: Executor multi has 199 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 199 parsl.dataflow.strategy DEBUG: Executor multi has 199 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 629f65719dfc664b4968ab4b7d15dd53 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8d4a57bf-5769-4c45-8a04-9d65ee32963e_calibrate_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8d4a57bf-5769-4c45-8a04-9d65ee32963e_calibrate_1039909_19.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a1e4a20ba033b2b7f8c4765164d8ff with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9e721341-14b4-49b4-8445-32bf3b616166_calibrate_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/9e721341-14b4-49b4-8445-32bf3b616166_calibrate_1039908_35.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec00d14842d96fc5b436b7ebc0985ada with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 59c3bad81fa0ffcdc6ecf378f6347f21 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/5402af71-58e6-48c5-8e13-22a3f485d903_calibrate_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/5402af71-58e6-48c5-8e13-22a3f485d903_calibrate_1040013_45.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/81101fd4-f1db-4ae8-94b3-ac06f90e6c5d_characterizeImage_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/81101fd4-f1db-4ae8-94b3-ac06f90e6c5d_characterizeImage_1040013_45.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8da54a12e67d8677c418bfdf5da21ffe with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f441c91d-3acb-4cb0-b048-8fee4fbb8e39_calibrate_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f441c91d-3acb-4cb0-b048-8fee4fbb8e39_calibrate_1039909_30.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa7ef0bb656e045455766fadc5905c6 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 9c56dfd42146a0691d270b9bbca47e74 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7aa17a7f-e391-4a00-b0e6-c87a6c7044e8_calibrate_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7aa17a7f-e391-4a00-b0e6-c87a6c7044e8_calibrate_1039909_20.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f4cf2294-8e39-40ca-8529-ef0efb2969f7_characterizeImage_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/f4cf2294-8e39-40ca-8529-ef0efb2969f7_characterizeImage_1039909_20.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff59b63c1e90e808ac31f21106a41ba with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a4c9c0a0-d89f-40d7-9e43-c9990a031051_calibrate_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a4c9c0a0-d89f-40d7-9e43-c9990a031051_calibrate_1039909_7.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e0536c29fddf10fb4d4112dcd081bf with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 55f148eee093d8d37d44e21df25169eb parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b10e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/46c12997-e573-494d-99b5-b987f716d048_calibrate_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/46c12997-e573-494d-99b5-b987f716d048_calibrate_1040013_48.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/602c49b0-6055-4210-a7b1-d2202bebb74f_characterizeImage_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/602c49b0-6055-4210-a7b1-d2202bebb74f_characterizeImage_1040013_48.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd22509240fd14bd7d8ffd19cc60da77 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a211025b-2abd-400e-91f7-47bcd97420ca_calibrate_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a211025b-2abd-400e-91f7-47bcd97420ca_calibrate_1039908_11.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d3922946db74ed5a528e3ca9e1a2740 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f111a4d1-23cc-4a78-9906-7c373dda5c12_calibrate_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f111a4d1-23cc-4a78-9906-7c373dda5c12_calibrate_1039908_17.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eec0e48a4374508561c8fcdfbf77c78 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/00cc2a20-38c9-40c3-bb2f-41d45f6b0640_calibrate_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/00cc2a20-38c9-40c3-bb2f-41d45f6b0640_calibrate_1039909_24.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d32ced14ceccef0c9d6564abde81ac5b with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/88d08a47-e48e-4772-af9c-483e56087f90_calibrate_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/88d08a47-e48e-4772-af9c-483e56087f90_calibrate_1039908_27.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3596bb9244e313cb9194114287b8e64 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 52f8bd93f745dd2ec312228479b8ce42 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46225946-0396-4f2e-b14b-b5cda7a0547f_calibrate_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46225946-0396-4f2e-b14b-b5cda7a0547f_calibrate_1039908_4.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75f7f23f-4139-4ed1-84cc-078ec64510fd_characterizeImage_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039908/75f7f23f-4139-4ed1-84cc-078ec64510fd_characterizeImage_1039908_4.stderr 2024-03-18 05:25:25 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:25:25 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73c902705a959df64ac6f53dfd6d66b2 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b7ed9573-68c9-4a56-971d-588d3501b876_calibrate_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b7ed9573-68c9-4a56-971d-588d3501b876_calibrate_1039908_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:25 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:25 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:25 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d870f6fdea6d4cec99ee399734365c5 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 543f08c3cf9a2828ee039d0263e906f6 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1e511aa-c1d7-4b17-a1e6-469a4913aa89_calibrate_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1e511aa-c1d7-4b17-a1e6-469a4913aa89_calibrate_1040013_7.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5de2c631-0cec-49ec-9a86-fa2a47d73d61_characterizeImage_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/5de2c631-0cec-49ec-9a86-fa2a47d73d61_characterizeImage_1040013_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:26 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70e00e7899c2e45df53a2a99d8db55c with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7908610e-241f-4374-8811-f5941da55e74_calibrate_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7908610e-241f-4374-8811-f5941da55e74_calibrate_1040013_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:26 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:26 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7d3bef2302637d17a63a9c28419625e with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 3a606c86173d9f2f620c7a4b1ef277b9 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b6dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ea501e99-1299-4d15-85e9-3a057a21f5fb_calibrate_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ea501e99-1299-4d15-85e9-3a057a21f5fb_calibrate_1040013_47.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dea17800-d329-44d1-aa2e-82ceba233d51_characterizeImage_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/dea17800-d329-44d1-aa2e-82ceba233d51_characterizeImage_1040013_47.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b163c2f8fd189c966c044139b66be99d with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5063683f-d13f-401e-a5a6-a065ff1afc35_calibrate_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5063683f-d13f-401e-a5a6-a065ff1afc35_calibrate_1039909_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:27 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dec9f59b8db01ccaffed27dad9bc9551 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8122c7a0-4840-4160-aacd-06205b9afdec_calibrate_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8122c7a0-4840-4160-aacd-06205b9afdec_calibrate_1039908_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:27 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7165f83c3e0ad3fa7236bb43272fe852 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2ca0c8d6-cc8b-4229-a9c4-5dd38c843383_calibrate_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2ca0c8d6-cc8b-4229-a9c4-5dd38c843383_calibrate_1039909_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:27 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21a3e4b64984415c7859e1fce6472fd5 with result from task 32 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:28 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8b82ed8f-bf6f-4716-a813-49e7effa8dce_calibrate_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8b82ed8f-bf6f-4716-a813-49e7effa8dce_calibrate_1039909_29.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff21cecb0bccc841eb7af4cd7a78f5cf with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6783a328-897d-4676-9753-2124a6c20c58_calibrate_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6783a328-897d-4676-9753-2124a6c20c58_calibrate_1039909_62.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac7cd78f270b2e66ca393d06fbeafcb with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/4b60c4dd-7931-438b-8841-698bea5fa028_calibrate_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/4b60c4dd-7931-438b-8841-698bea5fa028_calibrate_1039909_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:28 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:29 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6bf69eef1b0ee55891640c72d3bf299 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/fe916077-a8de-486b-ad32-603380fdfc08_calibrate_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/fe916077-a8de-486b-ad32-603380fdfc08_calibrate_1039909_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:29 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:30 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18189e3bd1385f5e8c9a03168f56804 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/05547afc-4275-4668-9fd7-5d6dfcb81d30_calibrate_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/05547afc-4275-4668-9fd7-5d6dfcb81d30_calibrate_1039907_26.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbfa7b36296ba531580fa4ff2edaa627 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/90134e5a-71a0-408e-90cb-480f77050dc8_calibrate_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/90134e5a-71a0-408e-90cb-480f77050dc8_calibrate_1039907_44.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 167d6b88296de76d9986bef5fb8efdaf with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/666ae569-43e9-443f-8ea7-4ee377b7c746_calibrate_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/666ae569-43e9-443f-8ea7-4ee377b7c746_calibrate_1039909_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:30 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f81734600cc01e47b5fffeb844b2ae95 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e5f7afc0-d049-42ef-a515-15517542b689_calibrate_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e5f7afc0-d049-42ef-a515-15517542b689_calibrate_1039908_41.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a01a86da87fab94f9bfd5c04f6a84288 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/fcd10c40-e65f-46c4-8975-f4558349ccf4_calibrate_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/fcd10c40-e65f-46c4-8975-f4558349ccf4_calibrate_1039907_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:31 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4939c385784ba3cf0baab6ca394bd97c with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8c492599-81bc-4cae-8757-cc88e89e5913_calibrate_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8c492599-81bc-4cae-8757-cc88e89e5913_calibrate_1039909_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:31 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:31 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccf27632ccbe02807bb6b08788c121e5 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cac158b3-3cc6-4019-9d20-91cb8730ff7a_calibrate_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cac158b3-3cc6-4019-9d20-91cb8730ff7a_calibrate_1039909_1.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acbf42717b1655678615171e7864cf47 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/491b0e5e-adf0-4aa3-8444-f457c379cac5_calibrate_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/491b0e5e-adf0-4aa3-8444-f457c379cac5_calibrate_1039907_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:32 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:32 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:33 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc45de50e521c1e1fbbabca2a7d15a60 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/5bbd54bf-49bd-4d53-abde-171538de79a8_calibrate_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/5bbd54bf-49bd-4d53-abde-171538de79a8_calibrate_1039908_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:33 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c9e270830d3e7ec114c883f7091316 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2f774426-df53-4f76-9fd0-d5d9ffc94a4a_calibrate_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/2f774426-df53-4f76-9fd0-d5d9ffc94a4a_calibrate_1039909_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:33 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d37a95920bdcc4288466313acd5b3538 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dfb5e21a-872f-437f-a330-bff833f74e08_calibrate_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dfb5e21a-872f-437f-a330-bff833f74e08_calibrate_1039909_16.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa85ed70e2832931e42444fd42a96c9 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 6be50b0ff40dba3adc674f9ea5e90c76 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b13f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/58ec6339-8c68-4a4a-ab16-401559378de4_calibrate_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/58ec6339-8c68-4a4a-ab16-401559378de4_calibrate_1040013_42.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/82dc26f9-f453-4aec-803c-e55005c29863_characterizeImage_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/82dc26f9-f453-4aec-803c-e55005c29863_characterizeImage_1040013_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:34 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b9373b832ca20d1da45ccfa1ca4eb6b with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/41ab65f0-587e-477c-94a9-8292217c4169_calibrate_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/41ab65f0-587e-477c-94a9-8292217c4169_calibrate_1039907_40.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4199cf89d04f77a644e1aea98f960684 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dc8d32ca-4145-491f-a32d-0f6b21bf50ba_calibrate_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dc8d32ca-4145-491f-a32d-0f6b21bf50ba_calibrate_1039908_9.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db0d1f28889efb36393f11941809082 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b50499e9-439f-40f5-a94b-7681f9b390d5_calibrate_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b50499e9-439f-40f5-a94b-7681f9b390d5_calibrate_1039908_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:34 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08e615ce4334c9d2b57d66002b73284 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/97ab4760-ab68-470e-b7b7-112f265bc1b6_calibrate_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/97ab4760-ab68-470e-b7b7-112f265bc1b6_calibrate_1039908_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:35 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8033931dd7a24ecc781c70c6950fb54 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d0f31115-5122-4016-8e7e-9f50777184a9_calibrate_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d0f31115-5122-4016-8e7e-9f50777184a9_calibrate_1039908_25.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17fd65abd06d4bd673c5aa03c3598dae with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/85aaec65-eff7-475e-b756-7f6e5fc04c6f_calibrate_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/85aaec65-eff7-475e-b756-7f6e5fc04c6f_calibrate_1039908_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:35 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c11b0defe82c21c660ef58352b62f172 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8b8d7a39-25ae-4ef9-bace-1f4c0c66d4d3_calibrate_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/8b8d7a39-25ae-4ef9-bace-1f4c0c66d4d3_calibrate_1039908_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:35 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:36 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:36 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c69c20591454f00799604eeed836e1 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8a917eeb-eed1-4ed5-b977-25818aa6e192_calibrate_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/8a917eeb-eed1-4ed5-b977-25818aa6e192_calibrate_1039909_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:37 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76313b6ba76797290ad5eadd86d44ffb with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/126964bf-7520-49a6-aae1-9a063f71d1ef_calibrate_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/126964bf-7520-49a6-aae1-9a063f71d1ef_calibrate_1039909_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:37 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry badd635e1bfccd55f9975e9018a0706d with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 0060efa8729b17e381931db467451f6d parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f69bf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/194b490d-3708-456c-8058-6374d774b7ef_calibrate_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/194b490d-3708-456c-8058-6374d774b7ef_calibrate_1040013_35.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ece9f156-d77d-4665-bd92-56582f57d33b_characterizeImage_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/ece9f156-d77d-4665-bd92-56582f57d33b_characterizeImage_1040013_35.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcd5d80a1ad9ff558cc8fbacb92474a4 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d233ac97-bac1-4103-9a34-20fa44b2ad9d_calibrate_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d233ac97-bac1-4103-9a34-20fa44b2ad9d_calibrate_1039907_57.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1feed64ee237b809e45d505f3c703c92 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/07c67ede-1cdc-4c7d-9ecf-109f52702422_calibrate_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/07c67ede-1cdc-4c7d-9ecf-109f52702422_calibrate_1039909_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:38 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af23cfb8e0309e3a388c1d9fb7cc507b with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/202761d5-5230-42f4-b396-b455bbd4e7ee_calibrate_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/202761d5-5230-42f4-b396-b455bbd4e7ee_calibrate_1039908_44.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56870bcdf9572daf3f85c649d51fcc9d with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f3c9eeee-24b8-468a-903e-1ab463032d02_calibrate_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f3c9eeee-24b8-468a-903e-1ab463032d02_calibrate_1039908_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:38 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:39 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeedb3168cf06b364abdead27477f994 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/099eb41e-705a-489a-9974-bc2e0fd38fa3_calibrate_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/099eb41e-705a-489a-9974-bc2e0fd38fa3_calibrate_1040013_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:39 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a582d5e7f41bcc83f1e1db3a74883fb8 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7a6c39a3-5d68-4fed-b466-c6b9dc8b32c3_calibrate_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/7a6c39a3-5d68-4fed-b466-c6b9dc8b32c3_calibrate_1039907_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:39 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:40 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a961e30e7fedc3acef972a46694a18e0 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 5fa041034ba597c24e81f1b3f1dd30e5 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6c0b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9f7928f-7028-4ef9-a9f0-6ccfcef8db7e_calibrate_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9f7928f-7028-4ef9-a9f0-6ccfcef8db7e_calibrate_1039909_10.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/df809a1c-4009-4129-881c-ebbfee8be267_characterizeImage_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1039909/df809a1c-4009-4129-881c-ebbfee8be267_characterizeImage_1039909_10.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d08e1be334259402b6630817c0bdac with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a24e61da-9361-4bc9-989b-cd47e13f08c8_calibrate_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/a24e61da-9361-4bc9-989b-cd47e13f08c8_calibrate_1039909_12.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a47de037823dd689e45a4068bbd150c3 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/808d3170-1a73-4ac8-9387-3bdcf05331a6_calibrate_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/808d3170-1a73-4ac8-9387-3bdcf05331a6_calibrate_1039907_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:40 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:41 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baab200a71902f9d448950caae17bee4 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2e5544f4-c4b8-4abb-93d2-d176dd7176c3_calibrate_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2e5544f4-c4b8-4abb-93d2-d176dd7176c3_calibrate_1039908_42.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd51b5f210597af9edec9047a6707642 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4d63c130-73ab-4720-9ee6-80dfce0ce712_calibrate_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4d63c130-73ab-4720-9ee6-80dfce0ce712_calibrate_1039908_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:41 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:41 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:42 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be05cad7b28649eba602e5087ddc5fd2 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f08919b8-671f-4c6c-92ae-b9f67fa6be12_calibrate_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/f08919b8-671f-4c6c-92ae-b9f67fa6be12_calibrate_1039908_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:42 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5108507f13bbd8f0838c5524716e9ab8 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3a4c1eeb-1099-4bb0-aa15-12d986a2fa93_calibrate_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/3a4c1eeb-1099-4bb0-aa15-12d986a2fa93_calibrate_1039908_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:42 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2cfabeb7d566caec1888e18803554eb with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dbc0be10-4dd4-4916-b0e0-e1336c842f8d_calibrate_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dbc0be10-4dd4-4916-b0e0-e1336c842f8d_calibrate_1039909_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:43 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e5b98c87db389d25d39588904945f6b with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b927927d-ea35-4ac4-baba-0b5ec477f09c_calibrate_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b927927d-ea35-4ac4-baba-0b5ec477f09c_calibrate_1039908_52.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73fa1c8d3c0bd227e66de629747ae72 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b3d32285-2a01-417d-9f69-3aea0a32429f_calibrate_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b3d32285-2a01-417d-9f69-3aea0a32429f_calibrate_1039909_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:43 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 740299c6f90f09d4dd9601e3351bc543 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/22062234-394a-4120-8f2a-d93e2d1d51d2_calibrate_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/22062234-394a-4120-8f2a-d93e2d1d51d2_calibrate_1039909_45.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebde6fd0f46a5741d64d7b0c1a7b530 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0ad27ecf-385f-4211-983a-32f66d82f01d_calibrate_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0ad27ecf-385f-4211-983a-32f66d82f01d_calibrate_1039909_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:43 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91caa66fc3160a644afa0d99d353607a with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4126f252-8523-4617-8090-c9736892b27e_calibrate_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4126f252-8523-4617-8090-c9736892b27e_calibrate_1040013_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:44 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:44 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7154d53060f4b849500ca6c168951074 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d91f7ab6-5870-41c5-a1de-982f7dbb2bba_calibrate_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d91f7ab6-5870-41c5-a1de-982f7dbb2bba_calibrate_1039908_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:44 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:45 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f2f4df4d0afa1d91b90407e15c1efe with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f32044e4-c0ff-4092-ac5b-2419eb7c86ef_calibrate_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f32044e4-c0ff-4092-ac5b-2419eb7c86ef_calibrate_1039907_51.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99ca5516e9dd22e8e01cc4fc35acda65 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/21bbbd1a-807d-465f-8686-f6db6b9cc7ab_calibrate_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/21bbbd1a-807d-465f-8686-f6db6b9cc7ab_calibrate_1039909_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:45 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:46 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:46 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:47 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:47 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c647879cc73a13df20ab05d49878bc66 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c4c5975e-9b9f-4988-add0-7a0a10477d0a_calibrate_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c4c5975e-9b9f-4988-add0-7a0a10477d0a_calibrate_1039907_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:47 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4347055ef00e5dbdac3b202c9ae29b5b with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d5334784-b661-4f49-989a-2e61dac03c99_calibrate_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d5334784-b661-4f49-989a-2e61dac03c99_calibrate_1039909_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:48 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20b0bcf76f293b4fb34500e78b7187f with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9f2f92b7-565e-4aff-8e1d-fdbd023ba8d9_calibrate_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9f2f92b7-565e-4aff-8e1d-fdbd023ba8d9_calibrate_1039907_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:48 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 138 parsl.dataflow.strategy DEBUG: Executor multi has 138 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 138 parsl.dataflow.strategy DEBUG: Executor multi has 138 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30b18881948272cc5d8c02c866c294c5 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3c346e06-9cbb-4ea7-83a0-f14e42a6ca8c_calibrate_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3c346e06-9cbb-4ea7-83a0-f14e42a6ca8c_calibrate_1039907_60.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 234cda71e4012baff51f853f4bcbb55e with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/063992a3-e304-4c8b-8833-30849060e9cb_calibrate_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/063992a3-e304-4c8b-8833-30849060e9cb_calibrate_1039907_37.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c405cbe0b92ef1da43c309b2dadc1db0 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d68a9427-bc5b-4d1f-a84c-a53cbd798113_calibrate_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d68a9427-bc5b-4d1f-a84c-a53cbd798113_calibrate_1039907_43.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc29bbc1424f55a66b73eb8d20d094d2 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/72c08a90-4828-48ca-b554-ac47b4b14c39_calibrate_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/72c08a90-4828-48ca-b554-ac47b4b14c39_calibrate_1039909_35.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618ab32f736b18e4ac9b71f1d6936f41 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dba19710-2767-4f13-8808-3f916927e44f_calibrate_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/dba19710-2767-4f13-8808-3f916927e44f_calibrate_1039909_32.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2db4ad72b6ee0d9350db8cb9070fede with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f8e4eb86-3d02-4c2b-b254-e21aab7cb90c_calibrate_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f8e4eb86-3d02-4c2b-b254-e21aab7cb90c_calibrate_1040013_11.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24f263a38a5e8a423e7d1db07f3088bb with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/65853974-0320-4552-9996-a96c33119abb_calibrate_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/65853974-0320-4552-9996-a96c33119abb_calibrate_1039907_10.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97f67339663d93cbe3603b8018fe5dba with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e2ddada-bea7-417e-b765-c53b7f00e631_calibrate_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e2ddada-bea7-417e-b765-c53b7f00e631_calibrate_1040013_56.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9eb05e3c02050d9204396ffa0f1c177 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/6fb71e44-0ca8-4f40-a2d1-089c8d28dfe1_calibrate_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/6fb71e44-0ca8-4f40-a2d1-089c8d28dfe1_calibrate_1040013_8.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e602a1619198cfecd730c273e517edf with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 9e2211be4d7e305638866f9691a30749 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f6b11bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/85e2c26d-2636-48ee-89a8-f92411751c1c_calibrate_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/85e2c26d-2636-48ee-89a8-f92411751c1c_calibrate_1040013_19.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e465a2a6-2fa6-4203-8227-6819d2acd508_characterizeImage_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/e465a2a6-2fa6-4203-8227-6819d2acd508_characterizeImage_1040013_19.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fee519dee78c3503091fa751f396c427 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0d217f02-a3a6-4f2e-add3-d825f5a3fae7_calibrate_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0d217f02-a3a6-4f2e-add3-d825f5a3fae7_calibrate_1040013_31.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 128 parsl.dataflow.strategy DEBUG: Executor multi has 128 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ded2a93f5412636c3ebeadfdda45ea with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9e958cd-7ad6-446a-bdc4-769664f19ee4_calibrate_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9e958cd-7ad6-446a-bdc4-769664f19ee4_calibrate_1039909_17.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b506ab1d983c00d052abfe0f358612 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/03a43b33-eec0-4598-9618-83870cb4b8c6_calibrate_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/03a43b33-eec0-4598-9618-83870cb4b8c6_calibrate_1039908_37.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8383efaeae44b2bd2ecd24bc28361e23 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/68c1060c-d59d-4bb8-a7a2-d626fcc97197_calibrate_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/68c1060c-d59d-4bb8-a7a2-d626fcc97197_calibrate_1039909_27.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58ef47f6e60f754d42f87c7183e6e9d with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/6372c030-5c17-4750-9add-ff42cadd7c49_calibrate_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/6372c030-5c17-4750-9add-ff42cadd7c49_calibrate_1039907_14.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10166c8a8bede1da0324d9e5fc0d3534 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2c0e29c5-aa02-4b58-9582-9e200eb3ee77_calibrate_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2c0e29c5-aa02-4b58-9582-9e200eb3ee77_calibrate_1039907_19.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a4367ffbf8e8eabd49f841c35e1405c with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/de35ef51-b4d3-4e25-8a80-df5fe68ed562_calibrate_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/de35ef51-b4d3-4e25-8a80-df5fe68ed562_calibrate_1039909_8.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fac201175ca911dd72a663384f077ac with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9a62e1ab-e713-4b91-b3d1-a4ef8bf4fd94_calibrate_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/9a62e1ab-e713-4b91-b3d1-a4ef8bf4fd94_calibrate_1039907_36.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91a07b10b845a140d3b1fd176a8baa0 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/82680853-d121-47d6-810b-6c4aa2d76ca5_calibrate_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/82680853-d121-47d6-810b-6c4aa2d76ca5_calibrate_1040013_24.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da82c421a4107cc38b5ef1585fc958ba with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b32153dc-ac18-4aa2-bbf6-c5749c4d0031_calibrate_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/b32153dc-ac18-4aa2-bbf6-c5749c4d0031_calibrate_1039908_57.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e149be91941dc49101b72f10feca158 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/171cb9d0-f9a7-4fa2-9ee5-cbc5664aaf34_calibrate_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/171cb9d0-f9a7-4fa2-9ee5-cbc5664aaf34_calibrate_1039908_58.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f4f29957cc465de1d8ec5e410ff2587 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46c54c81-afe5-4d4f-899b-d56b2b8bdd1f_calibrate_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46c54c81-afe5-4d4f-899b-d56b2b8bdd1f_calibrate_1039908_10.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26a432797ec90c065b3f523779e4cf16 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/03627040-5583-4a9e-b96b-c4eda1cf78e7_calibrate_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/03627040-5583-4a9e-b96b-c4eda1cf78e7_calibrate_1040013_34.stderr 2024-03-18 05:25:59 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:25:59 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82568e021314a6471718f0cb691e8792 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f6081528-0391-4662-88ee-ddd516557e8a_calibrate_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/f6081528-0391-4662-88ee-ddd516557e8a_calibrate_1039907_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:25:59 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:00 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:00 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7ea18103e399550f8baa6e44e2516d with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/ebfef480-995b-4f56-9c46-c364ebd9f6b6_calibrate_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/ebfef480-995b-4f56-9c46-c364ebd9f6b6_calibrate_1039907_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:01 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9554e7a144b3084d35d4a969784af54c with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/51630cb9-da03-41d6-9760-0e29b8bb6b5d_calibrate_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/51630cb9-da03-41d6-9760-0e29b8bb6b5d_calibrate_1040013_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:01 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df3898a8696e330e8918597342e22bd5 with result from task 509 parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:01 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5ced387f-55cc-4054-a3bc-64dc25451625_calibrate_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5ced387f-55cc-4054-a3bc-64dc25451625_calibrate_1039909_57.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a2959a51d55b7773100cbcb99a8fcae with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b4bbe2e4-81a6-431b-a64f-6541853b0940_calibrate_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b4bbe2e4-81a6-431b-a64f-6541853b0940_calibrate_1039909_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:02 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9c721ec9d290c01914a894ad95b9f44 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/55ee9186-02d5-4617-a1a7-1b06548da7e7_calibrate_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/55ee9186-02d5-4617-a1a7-1b06548da7e7_calibrate_1039907_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:02 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f59a3b351509e79c9f62f96885136e51 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/51d82ac5-d4ef-4e8f-8459-0ea0b4e507af_calibrate_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/51d82ac5-d4ef-4e8f-8459-0ea0b4e507af_calibrate_1039909_41.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62953e95001673fbfb5b7d7c7dccb1c9 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d314df02-75d4-4b9b-adf0-eb2f046284bb_calibrate_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d314df02-75d4-4b9b-adf0-eb2f046284bb_calibrate_1039907_53.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf654d12d83c8aba95650d4126c5f49 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/19892956-aacc-4de8-830b-530ae1cf47ba_calibrate_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/19892956-aacc-4de8-830b-530ae1cf47ba_calibrate_1039907_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:03 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 071e4a13d71ec6861f462bb40ecf4718 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/73117f83-7aa2-4856-9e15-c037f71ab8b0_calibrate_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/73117f83-7aa2-4856-9e15-c037f71ab8b0_calibrate_1040013_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:03 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a73a7e8acb703cc403bdacd8caa0ff9b with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e7c00cef-00e2-4686-8ac6-e6c7d1732e40_calibrate_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e7c00cef-00e2-4686-8ac6-e6c7d1732e40_calibrate_1039908_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:04 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca893badbf9b7d20f37d5b5eef00609b with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/85661529-1e7e-4329-abbf-dfe041de6048_calibrate_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/85661529-1e7e-4329-abbf-dfe041de6048_calibrate_1039907_30.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13694b0518ef9f55898e4d625bc9b055 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c009b0b6-0710-4079-a54a-5d760b83912c_calibrate_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c009b0b6-0710-4079-a54a-5d760b83912c_calibrate_1039909_22.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 116397f9e38f2b05a5190f33806808b8 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d03c70f4-4664-4082-9045-ce8d932f45b3_calibrate_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/d03c70f4-4664-4082-9045-ce8d932f45b3_calibrate_1039909_38.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9a0e31116e8108b1f3a5a66d95fbc3 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/99d81a2c-efe0-4972-bb53-d250b8e1e73b_calibrate_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/99d81a2c-efe0-4972-bb53-d250b8e1e73b_calibrate_1039907_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:05 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:05 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c07df91dbe299f7b40fa9eac549553 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cb0a127d-9fe3-4406-947d-c188ab31b1e0_calibrate_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cb0a127d-9fe3-4406-947d-c188ab31b1e0_calibrate_1039909_26.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f4822245c892a6e69e9e573838a76cf with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4154cdd0-6eeb-47c1-ba58-83f46eeb6146_calibrate_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/4154cdd0-6eeb-47c1-ba58-83f46eeb6146_calibrate_1039908_45.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5bcb1736f6423fdb9a9737dc0945129 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bd9147a-99de-4aab-92e3-4f52154f35a4_calibrate_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bd9147a-99de-4aab-92e3-4f52154f35a4_calibrate_1039909_6.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15a1bbdc081326647387cff412e571d8 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1f70dde3-c1d3-479b-9e03-94bdc0dcbe43_calibrate_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1f70dde3-c1d3-479b-9e03-94bdc0dcbe43_calibrate_1039907_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:06 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd596403af9aaf8f7cb2078b8c3e5f03 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/662a84f1-cee2-4c22-a753-2cdfcafd72fd_calibrate_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/662a84f1-cee2-4c22-a753-2cdfcafd72fd_calibrate_1039907_21.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f9b410a537049987dd8a0915bec513 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ed549143-0dea-4576-9c9d-64fcd383fde3_calibrate_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ed549143-0dea-4576-9c9d-64fcd383fde3_calibrate_1039909_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:06 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69fefc0e46e3365298bb8aa9b0f4d49 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0e67f632-8b26-4de3-9d4f-6796754d3198_calibrate_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0e67f632-8b26-4de3-9d4f-6796754d3198_calibrate_1039909_5.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7881124e11fe86a686b45666c7670e7 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9cd884fe-ff90-4a8f-ad09-8e0bc3d654e8_calibrate_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9cd884fe-ff90-4a8f-ad09-8e0bc3d654e8_calibrate_1040013_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:07 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:07 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 675a95ef02a0db1d82bf209ce79ec2da with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bf2b3a9d-aff8-4d2d-b386-f2f3c5955511_calibrate_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/bf2b3a9d-aff8-4d2d-b386-f2f3c5955511_calibrate_1039908_7.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f39d256db81029ddda15580962c7097 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ba77b11e-e1fd-4884-9192-ceec1867afb8_calibrate_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ba77b11e-e1fd-4884-9192-ceec1867afb8_calibrate_1040013_38.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29a31db3e6303607a1cc18fd4bd448db with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/83898f83-71f1-4350-adfe-aab9e31321d1_calibrate_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/83898f83-71f1-4350-adfe-aab9e31321d1_calibrate_1039907_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:08 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f48d05e5ec4f42ef131b8ea0742b2b50 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d11e184d-0e73-440e-8565-88d5696749a8_calibrate_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/d11e184d-0e73-440e-8565-88d5696749a8_calibrate_1039908_19.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9723e63d517050bfd8c466e9fd88bb5 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3b52e5cc-1d60-481a-a406-7a112c104d4e_calibrate_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3b52e5cc-1d60-481a-a406-7a112c104d4e_calibrate_1039907_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:08 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8790ff5c098d062c45363a4800bd8b with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ae47bc07-150e-4ee8-909d-0530c0a01e01_calibrate_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/ae47bc07-150e-4ee8-909d-0530c0a01e01_calibrate_1039909_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:09 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 641203232fed82915d1241c4d1d29fac with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f2fe57b1-82fa-4f1d-ad42-ee15ceb63f40_calibrate_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/f2fe57b1-82fa-4f1d-ad42-ee15ceb63f40_calibrate_1039909_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:09 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2108594064fc9db884a6dd08dfd3885 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0815ce54-17bf-480d-bac6-28129d2c295e_calibrate_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/0815ce54-17bf-480d-bac6-28129d2c295e_calibrate_1040013_40.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41448fc068e67aa4b47d2055d4e15cb7 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a0515838-4a34-4495-8ed0-06d3701955b8_calibrate_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/a0515838-4a34-4495-8ed0-06d3701955b8_calibrate_1039907_56.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe11c4ccc2af570a617fafd8e3d04136 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/3d250521-2ba8-4c25-ad1a-57692af79979_calibrate_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/3d250521-2ba8-4c25-ad1a-57692af79979_calibrate_1040013_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:10 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e696c0e4047435c0ad543d2ce7f8575 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2a5e0a77-0ac9-4148-8c53-a859288249f2_calibrate_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2a5e0a77-0ac9-4148-8c53-a859288249f2_calibrate_1039907_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:10 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 607d0a65c1e03595254b017389db5dca with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c1a2eadb-ba25-4fb9-a48c-675341fc4407_calibrate_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/c1a2eadb-ba25-4fb9-a48c-675341fc4407_calibrate_1039907_18.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5dc034a14a76403c4f8f6696d2d4744 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cfae7756-588a-4567-8d25-30a737fd0778_calibrate_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/cfae7756-588a-4567-8d25-30a737fd0778_calibrate_1039909_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:11 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:11 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:11 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f48de504d9beb0119d7b770e6b9e09 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0276b19-dc45-4776-b93f-1dbf7e5e5ddf_calibrate_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0276b19-dc45-4776-b93f-1dbf7e5e5ddf_calibrate_1040013_25.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ca487d1ae7283eba4f8459fc07e7c12 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/15abf8ea-2e50-4ec6-89d7-c134410de001_calibrate_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/15abf8ea-2e50-4ec6-89d7-c134410de001_calibrate_1040013_51.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01998b416066a429b5dd7eb94c977461 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2fe9b871-3e46-457a-beed-94a25b1c71ce_calibrate_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2fe9b871-3e46-457a-beed-94a25b1c71ce_calibrate_1040013_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:12 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154231b9630d337fad121183e4ad97d5 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/33404284-9fe8-4334-9f91-a89b4b1030a4_calibrate_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/33404284-9fe8-4334-9f91-a89b4b1030a4_calibrate_1040013_3.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ebd8b80e2c3e3d3467ae3438f8f1a13 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0f0ae67-3d9a-4efc-9fc1-30241c46b71c_calibrate_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b0f0ae67-3d9a-4efc-9fc1-30241c46b71c_calibrate_1040013_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:12 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c82b0d7c431827b66dd0481492169dd0 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bde7434-af15-42f9-8fe8-86ad043b1fed_calibrate_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/5bde7434-af15-42f9-8fe8-86ad043b1fed_calibrate_1039909_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:12 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 310c6f37b2ea533708fee4ed67471b6c with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0d798f34-f2e7-452a-9946-3f10bf15903d_calibrate_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0d798f34-f2e7-452a-9946-3f10bf15903d_calibrate_1039909_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:13 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2eaa31975afcbde0e22e3209726e9e with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3d686b17-1995-4943-bebd-fa7f16e11e8f_calibrate_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/3d686b17-1995-4943-bebd-fa7f16e11e8f_calibrate_1039907_17.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f25502627b7a80d220165a28a08b4b4 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d615a00d-9784-4509-a5a4-a5eb2c950762_calibrate_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d615a00d-9784-4509-a5a4-a5eb2c950762_calibrate_1039907_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:13 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:13 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2c5f55831bea40b7227ee7ce3b2b623 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1fdff434-03bf-443e-b985-25a11dd65eff_calibrate_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1fdff434-03bf-443e-b985-25a11dd65eff_calibrate_1039908_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245469972203fcbc1711e082a8e0cdaa with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2e3c5d53-22d2-4147-9cac-c9b963cfe9d3_calibrate_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2e3c5d53-22d2-4147-9cac-c9b963cfe9d3_calibrate_1039907_6.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ddd25d3fbac63287806597f73ebb8be with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a2f28eb7-fb19-4593-93b2-3cb5432b1ed1_calibrate_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/a2f28eb7-fb19-4593-93b2-3cb5432b1ed1_calibrate_1039908_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85df5046316f39e6082ea021893c19e9 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/60a533d0-22bd-45d4-acc7-99f0643e4a0b_calibrate_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/60a533d0-22bd-45d4-acc7-99f0643e4a0b_calibrate_1039907_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:14 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45c08d47daf76e7e1e2c06d6b690f334 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2a914ba5-d692-45fa-941f-51062b43e07c_calibrate_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/2a914ba5-d692-45fa-941f-51062b43e07c_calibrate_1040013_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:15 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379d3985fc0e94b3a7f70339baf9832e with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4f79d61f-bb4e-4c61-ba10-4c8081cd5866_calibrate_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4f79d61f-bb4e-4c61-ba10-4c8081cd5866_calibrate_1040013_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f63a6ce2f5a81a4c6480fa28d9c70ce with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2d55b173-f666-4825-bd42-a7559fc2ad90_calibrate_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/2d55b173-f666-4825-bd42-a7559fc2ad90_calibrate_1039908_21.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7573fb4b4123d98d3c7f90579ea2c661 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dbad4805-54ae-4349-a6bb-c33a87ddc87c_calibrate_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/dbad4805-54ae-4349-a6bb-c33a87ddc87c_calibrate_1039908_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:16 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:17 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:17 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:17 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0cc526aeb248070e702275970c3be1 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1a0098d9-be2f-4729-be0c-4574e82cca92_calibrate_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1a0098d9-be2f-4729-be0c-4574e82cca92_calibrate_1039908_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:17 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:17 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 62 parsl.dataflow.strategy DEBUG: Executor multi has 62 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 62 parsl.dataflow.strategy DEBUG: Executor multi has 62 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ab8bfe689aa2586d769b1b60e32527 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0b2587cd-5c07-4464-8139-2218b3c51d7a_calibrate_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/0b2587cd-5c07-4464-8139-2218b3c51d7a_calibrate_1039909_60.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e3256f9a1edafabfe337169d1270619 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/efe658b2-8fcc-4db6-8337-4f8ddc8b590b_calibrate_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/efe658b2-8fcc-4db6-8337-4f8ddc8b590b_calibrate_1039908_48.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 252fe5c91d12dd90142baa024521fa9f with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0208a3bf-bf62-4f66-8bbf-7ae131ed6aa8_calibrate_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0208a3bf-bf62-4f66-8bbf-7ae131ed6aa8_calibrate_1039908_46.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 252f38fd22acb2dde3c667378cd2a332 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ff697bc2-7f16-4bf7-acc9-11b9e10816a1_calibrate_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ff697bc2-7f16-4bf7-acc9-11b9e10816a1_calibrate_1040013_57.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52ed69d36f840b7e9514b362e007082e with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e67a5906-736a-4379-b9fe-b0794c01d2da_calibrate_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/e67a5906-736a-4379-b9fe-b0794c01d2da_calibrate_1039908_62.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c8c735bc426ea381e6d22a57502b56 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2d35a1fa-ccd1-4946-bcd6-887ce110cf79_calibrate_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/2d35a1fa-ccd1-4946-bcd6-887ce110cf79_calibrate_1039907_5.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65ba1cbd222b40e2e1280f2f1eb0c116 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d2e967da-1d50-4494-baaf-ce2837757aaf_calibrate_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d2e967da-1d50-4494-baaf-ce2837757aaf_calibrate_1039907_24.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac7bfc6e309a4502684d085bc23fa060 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/485ea2dd-ab41-49d7-a897-62a26fd9184e_calibrate_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/485ea2dd-ab41-49d7-a897-62a26fd9184e_calibrate_1040013_18.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5113c3406d8660604fa858d9e9199194 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/20ec4892-682f-4392-a3e6-3735b5147bba_calibrate_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/20ec4892-682f-4392-a3e6-3735b5147bba_calibrate_1039908_26.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf3ba8c766673f3fab48aaa2a1851fa with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/daa8b94b-3951-4934-a18f-1b14cba22352_calibrate_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/daa8b94b-3951-4934-a18f-1b14cba22352_calibrate_1040013_23.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbcc5d55e2602bedf2866cb06ade0d9d with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1fe43ad2-e16f-48ef-ac51-5d979e8b92f3_calibrate_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/1fe43ad2-e16f-48ef-ac51-5d979e8b92f3_calibrate_1039907_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 51 parsl.dataflow.strategy DEBUG: Executor multi has 51 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005b5ac1da93bfcc2c65fa52630d6caa with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b4b47508-26b5-4be0-9288-42869aed04c1_calibrate_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b4b47508-26b5-4be0-9288-42869aed04c1_calibrate_1040013_9.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61f86ee3d5903451f3ed3fb5a87177b4 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/39739fb8-7aea-4a94-b905-d4cb8235c5b4_calibrate_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/39739fb8-7aea-4a94-b905-d4cb8235c5b4_calibrate_1040013_17.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14072301b003caa4247018196bccf9da with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6cdfa6e8-7649-4e10-a9fd-8cb07648de13_calibrate_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/6cdfa6e8-7649-4e10-a9fd-8cb07648de13_calibrate_1039909_59.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c0f54805969d52718ad437a98a4eaf with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/06dd8328-c5c5-4dd5-9c20-9bcd949592db_calibrate_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/06dd8328-c5c5-4dd5-9c20-9bcd949592db_calibrate_1039909_34.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf4e1cc265d10584136339ea31a839d with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d6fefe0b-b9af-44f8-9f99-d7d945641320_calibrate_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039907/d6fefe0b-b9af-44f8-9f99-d7d945641320_calibrate_1039907_16.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bee0e1756deae48ab6dab17f0349459 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/cb0458d9-ac2b-40c6-911d-df813e4abb4a_calibrate_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/cb0458d9-ac2b-40c6-911d-df813e4abb4a_calibrate_1039908_23.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55b06b562d02fc413903fe287db1c18c with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/abf3e5b5-5759-4579-b507-800dbbed7547_calibrate_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/abf3e5b5-5759-4579-b507-800dbbed7547_calibrate_1040013_13.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2def4f85d1c7a5217da0339a85b61b06 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c6d4c696-bf76-4db5-96ac-0b30a4a8c276_calibrate_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/c6d4c696-bf76-4db5-96ac-0b30a4a8c276_calibrate_1039908_30.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 43 parsl.dataflow.strategy DEBUG: Executor multi has 43 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bddef693f3d07632765b47eaf36906a4 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1be981d7-6cad-4ca0-bd06-336f063ff7d1_calibrate_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/1be981d7-6cad-4ca0-bd06-336f063ff7d1_calibrate_1039908_32.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e846cac7651566077c4924fee15f0a28 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/685f19d1-fabb-4b50-8287-c037e75975d3_calibrate_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/685f19d1-fabb-4b50-8287-c037e75975d3_calibrate_1039908_6.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72be6b8aca4e99b606c57cfe819dd437 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/18421d18-a8dd-4ff7-abce-3a91dea7c1ed_calibrate_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/18421d18-a8dd-4ff7-abce-3a91dea7c1ed_calibrate_1040013_14.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bfb843a4786bd26dc42faa048558c90 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7d175e88-f3e1-4d98-9423-1c8a9fa697da_calibrate_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7d175e88-f3e1-4d98-9423-1c8a9fa697da_calibrate_1039909_11.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70899772ca08597dbed844bd117e0d7 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4335f9a0-59c7-463a-b257-4a8ee5127608_calibrate_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/4335f9a0-59c7-463a-b257-4a8ee5127608_calibrate_1040013_30.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca5dafb4fecfb9825a0829bf8279892 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c691cf37-795b-4fef-9e11-e7d33d290da4_calibrate_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/c691cf37-795b-4fef-9e11-e7d33d290da4_calibrate_1039909_55.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9770e831e4cc8359f0cfcc0e17401fc with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/1ebc82b6-7660-4ac5-b42b-f928f8e09105_calibrate_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/1ebc82b6-7660-4ac5-b42b-f928f8e09105_calibrate_1039909_49.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d70b2975784cd5e9519a8776e603af77 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0337acb0-fd1c-4736-ac9b-1d81d149f66c_calibrate_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/0337acb0-fd1c-4736-ac9b-1d81d149f66c_calibrate_1039908_53.stderr 2024-03-18 05:26:32 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:26:32 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:32 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:33 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66762972e7aa7771d09be295d0f42e16 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1700267-aba9-4e73-b261-729886aa4095_calibrate_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1700267-aba9-4e73-b261-729886aa4095_calibrate_1040013_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe8f69a5b34d58d8db6f57c4f9de488 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fb606ff5-66e5-4f7e-be99-f0a95d11fc06_calibrate_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fb606ff5-66e5-4f7e-be99-f0a95d11fc06_calibrate_1040013_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:34 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61417c797e9606b77c0c9a16bb5b3f52 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/153114a1-71db-42a8-94d6-654eb06b8d72_calibrate_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/153114a1-71db-42a8-94d6-654eb06b8d72_calibrate_1040013_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c56dfd42146a0691d270b9bbca47e74 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7aa17a7f-e391-4a00-b0e6-c87a6c7044e8_calibrate_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/7aa17a7f-e391-4a00-b0e6-c87a6c7044e8_calibrate_1039909_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:35 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:36 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f8bd93f745dd2ec312228479b8ce42 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46225946-0396-4f2e-b14b-b5cda7a0547f_calibrate_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/46225946-0396-4f2e-b14b-b5cda7a0547f_calibrate_1039908_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7262d89aebe56f8e394089f1ca8517be with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/86baaf97-ad96-436b-b44a-a539ad1617c2_calibrate_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/86baaf97-ad96-436b-b44a-a539ad1617c2_calibrate_1040013_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:37 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa041034ba597c24e81f1b3f1dd30e5 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9f7928f-7028-4ef9-a9f0-6ccfcef8db7e_calibrate_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039909/b9f7928f-7028-4ef9-a9f0-6ccfcef8db7e_calibrate_1039909_10.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e6e745fffee4588860a6356f04a8e26 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/67577e02-6184-483d-bbf4-431eb7eb35e4_calibrate_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1039908/67577e02-6184-483d-bbf4-431eb7eb35e4_calibrate_1039908_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a55cdcbd1376be4a03bc9b4741b8bdf with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b238722e-d4ff-46fd-933f-01bdc86c2966_calibrate_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/b238722e-d4ff-46fd-933f-01bdc86c2966_calibrate_1040013_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:38 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5d24e7026842c87fbb70c72b3b01d0 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f1afb7aa-5b5b-4fa1-83e7-7c6a1808a077_calibrate_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f1afb7aa-5b5b-4fa1-83e7-7c6a1808a077_calibrate_1040013_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:39 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:26:40 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 25 parsl.dataflow.strategy DEBUG: Executor multi has 25 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 25 parsl.dataflow.strategy DEBUG: Executor multi has 25 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 499fae0af570c0bf43803ae067a7466c with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f0dcfc1d-29a5-4402-b014-1d9195f63f7e_calibrate_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f0dcfc1d-29a5-4402-b014-1d9195f63f7e_calibrate_1040013_4.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdce20978fe045451a1a242a01555d7 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/14c8b6a8-61f1-4663-a409-682d359ecd7c_calibrate_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/14c8b6a8-61f1-4663-a409-682d359ecd7c_calibrate_1040013_37.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c147af9f5baf8dd85de25ae6617f1ae7 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/71e4d1a6-cd20-4414-82fd-94d3975b1d30_calibrate_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/71e4d1a6-cd20-4414-82fd-94d3975b1d30_calibrate_1040013_27.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f006f3cc282ded1c0c4b960030dec34b with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e2444c9e-c3ae-459a-b3d3-3cf7ec5f1338_calibrate_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e2444c9e-c3ae-459a-b3d3-3cf7ec5f1338_calibrate_1040013_60.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 782d85bde35a8f42cadaeb69f1c4aa6f with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e52c4cb4-6ce0-4bc4-81ea-e10b66136856_calibrate_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e52c4cb4-6ce0-4bc4-81ea-e10b66136856_calibrate_1040013_15.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b190a45fe1d145139286a40ad0664ee3 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e089132e-29e8-449e-a629-e309152a92d7_calibrate_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/e089132e-29e8-449e-a629-e309152a92d7_calibrate_1040013_5.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9732366702c5a4f80f6f218130ed1e9c with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f833ac08-4b42-41c4-9e40-2c89d8a84d3c_calibrate_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/f833ac08-4b42-41c4-9e40-2c89d8a84d3c_calibrate_1040013_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97bde07702e19d4521318c1e69395ed5 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/454aac97-d5be-4036-956b-8ee9618c9322_calibrate_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/454aac97-d5be-4036-956b-8ee9618c9322_calibrate_1040013_32.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f70877ee307150aaabaead19125703 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1e2198a0-6024-425c-b7c2-853c9be138d3_calibrate_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1e2198a0-6024-425c-b7c2-853c9be138d3_calibrate_1040013_52.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda0a3da51fc3f0404b79bffb61a50d2 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/42ac8433-4e76-4c64-aadb-763277a4850e_calibrate_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/42ac8433-4e76-4c64-aadb-763277a4850e_calibrate_1040013_26.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c48e4c46e9e7e208951f09f22eb053fa with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1bf3dfd0-a9a1-425e-a389-1f859ac7eca4_calibrate_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1bf3dfd0-a9a1-425e-a389-1f859ac7eca4_calibrate_1040013_36.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0060efa8729b17e381931db467451f6d with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/194b490d-3708-456c-8058-6374d774b7ef_calibrate_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/194b490d-3708-456c-8058-6374d774b7ef_calibrate_1040013_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 13 parsl.dataflow.strategy DEBUG: Executor multi has 13 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d650f7410b6121456955a6be58da9875 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fabe2178-90fc-4977-99bb-95549f6cf75f_calibrate_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/fabe2178-90fc-4977-99bb-95549f6cf75f_calibrate_1040013_49.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3db990a48c5167c8052198608f8b0106 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7ba7aa41-523e-46b6-a8b4-09a73cf5449e_calibrate_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/7ba7aa41-523e-46b6-a8b4-09a73cf5449e_calibrate_1040013_20.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7dfa482e0af876e3068ed34996eb977 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1a09c1c8-bb54-4c3b-8493-320cb12b39b9_calibrate_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/1a09c1c8-bb54-4c3b-8493-320cb12b39b9_calibrate_1040013_59.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5b8420765f3c6d2656599fc4c7e6c46 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a020c357-8fd1-4543-ad50-2100764d0c01_calibrate_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a020c357-8fd1-4543-ad50-2100764d0c01_calibrate_1040013_39.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed8762c2c351c812ba648ca356a4e7e with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e0eb1de-e0de-41a2-91c2-73a91020216f_calibrate_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9e0eb1de-e0de-41a2-91c2-73a91020216f_calibrate_1040013_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 543f08c3cf9a2828ee039d0263e906f6 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1e511aa-c1d7-4b17-a1e6-469a4913aa89_calibrate_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/a1e511aa-c1d7-4b17-a1e6-469a4913aa89_calibrate_1040013_7.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb416e1dd82f5ca64a584dac0493b8f0 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/36611d25-2576-41c8-8321-e2f3dca6ea48_calibrate_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/36611d25-2576-41c8-8321-e2f3dca6ea48_calibrate_1040013_53.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c3bad81fa0ffcdc6ecf378f6347f21 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/5402af71-58e6-48c5-8e13-22a3f485d903_calibrate_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/5402af71-58e6-48c5-8e13-22a3f485d903_calibrate_1040013_45.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55f148eee093d8d37d44e21df25169eb with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/46c12997-e573-494d-99b5-b987f716d048_calibrate_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/46c12997-e573-494d-99b5-b987f716d048_calibrate_1040013_48.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6be50b0ff40dba3adc674f9ea5e90c76 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/58ec6339-8c68-4a4a-ab16-401559378de4_calibrate_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/58ec6339-8c68-4a4a-ab16-401559378de4_calibrate_1040013_42.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:06 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e2211be4d7e305638866f9691a30749 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/85e2c26d-2636-48ee-89a8-f92411751c1c_calibrate_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/85e2c26d-2636-48ee-89a8-f92411751c1c_calibrate_1040013_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:07 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:08 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8770358a28e9262e067751a9fe9b8247 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 1e58cc6bbb982a3f3ce2f6ccb7441c72 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/8f470f7f-e7fd-4b6f-9af2-d4f5d640fe80_characterizeImage_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/characterizeImage/1040013/8f470f7f-e7fd-4b6f-9af2-d4f5d640fe80_characterizeImage_1040013_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:09 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a606c86173d9f2f620c7a4b1ef277b9 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ea501e99-1299-4d15-85e9-3a057a21f5fb_calibrate_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/ea501e99-1299-4d15-85e9-3a057a21f5fb_calibrate_1040013_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:10 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:11 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:12 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:13 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:14 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:14 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:39 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:40 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:41 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:42 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:43 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:44 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:45 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:46 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:47 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:47 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:47 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:27:47 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 0 failed parsl.dataflow.dflow INFO: Task 122 marked for retry parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 1e58cc6bbb982a3f3ce2f6ccb7441c72 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1505f8188900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 1 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 2024-03-18 05:28:12 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:28:12 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:12 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:12 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:13 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:14 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:15 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:16 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:17 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:18 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:19 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:20 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:46 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:47 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:48 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:49 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195092,17195093,17195094,17195095,17195096,17195097,17195098,17195099,17195100,17195101,17195102,17195103,17195104,17195106,17195107,17195108,17195109,17195110,17195111,17195112,17195113,17195114,17195115,17195116,17195117,17195118,17195119,17195120,17195122,17195123' parsl.providers.slurm.slurm DEBUG: squeue returned 17195123 R 17195112 R 17195113 R 17195114 R 17195115 R 17195116 R 17195117 R 17195118 R 17195119 R 17195120 R 17195122 R 17195107 R 17195108 R 17195109 R 17195110 R 17195111 R 17195106 R 17195096 R 17195097 R 17195098 R 17195099 R 17195100 R 17195101 R 17195102 R 17195103 R 17195104 R 17195092 R 17195093 R 17195094 R 17195095 R parsl.providers.slurm.slurm DEBUG: Updating job 17195123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195095 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195124,17195125,17195126,17195127,17195128' parsl.providers.slurm.slurm DEBUG: squeue returned 17195124 R 17195125 R 17195126 R 17195127 R 17195128 R parsl.providers.slurm.slurm DEBUG: Updating job 17195124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195128 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:50 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:51 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:52 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195130,17195131,17195132,17195133,17195134,17195135,17195136,17195137,17195138,17195139,17195140,17195141,17195142,17195143,17195144,17195145,17195146,17195147,17195148,17195149' parsl.providers.slurm.slurm DEBUG: squeue returned 17195149 R 17195143 R 17195144 R 17195145 R 17195146 R 17195147 R 17195148 R 17195140 R 17195141 R 17195142 R 17195130 R 17195131 R 17195132 R 17195133 R 17195134 R 17195135 R 17195136 R 17195137 R 17195138 R 17195139 R parsl.providers.slurm.slurm DEBUG: Updating job 17195149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195139 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:53 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:54 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:28:54 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195150,17195151,17195152,17195153,17195154,17195155' parsl.providers.slurm.slurm DEBUG: squeue returned 17195154 R 17195155 R 17195151 R 17195152 R 17195153 R 17195150 R parsl.providers.slurm.slurm DEBUG: Updating job 17195154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23115487136400 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 1 failed parsl.dataflow.dflow ERROR: Task 122 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1e58cc6bbb982a3f3ce2f6ccb7441c72 with result from task 122 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 729 parsl.dataflow.dflow INFO: Tasks in state States.failed: 8 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 7 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/submit/DEEP/20211003/B1b/science#step1/20240318T051950Z/logs/calibrate/1040013/9042ec8a-fed9-485b-87e9-4b817476f054_calibrate_1040013_61.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6c43022207e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6c43022207e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4fff0440e5be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4fff0440e5be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2261b84c7073 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2261b84c7073 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18cdb2d85df3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18cdb2d85df3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ee015c545a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ee015c545a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 23ad1dd93776 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 23ad1dd93776 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8da97d199c60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8da97d199c60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: de83d39e17f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: de83d39e17f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf5001e5beaf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf5001e5beaf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 936e563f1add parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 936e563f1add parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f3846f63747 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f3846f63747 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5eb3c93be5b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5eb3c93be5b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5091a4d55216 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5091a4d55216 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e3025fcf2133 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e3025fcf2133 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8200f7beb72d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8200f7beb72d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f4c286410bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f4c286410bb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97f7a073cd39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97f7a073cd39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f266da69716 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f266da69716 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8cb730971f4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8cb730971f4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d2fc12b6da2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d2fc12b6da2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 98b017b9794c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 98b017b9794c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dacbda480959 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dacbda480959 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9a553191baa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9a553191baa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2609d1cda5e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2609d1cda5e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31b97db8aff2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31b97db8aff2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e87345cbeb1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e87345cbeb1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 366a03930253 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 366a03930253 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e4bc5a1ed32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e4bc5a1ed32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e6c44e403773 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e6c44e403773 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1e9c4408659 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1e9c4408659 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b167d0e05836 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b167d0e05836 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4621e1ff410d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4621e1ff410d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0729e1813837 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0729e1813837 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fcc0e98bc282 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fcc0e98bc282 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43f0dc055782 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43f0dc055782 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ce4b00dab05 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ce4b00dab05 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1663082de7ad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1663082de7ad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c2a0cb89591 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c2a0cb89591 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9daac4fbd181 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9daac4fbd181 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8ca9eefa448 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8ca9eefa448 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c27a5a5d98e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c27a5a5d98e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 310b1e424cb9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 310b1e424cb9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba26a04582c6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba26a04582c6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ae8fe9144f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ae8fe9144f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3cbf86e95bc0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3cbf86e95bc0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da5949a389e7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da5949a389e7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 21f80ed65f42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 21f80ed65f42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70c06e24fcd9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70c06e24fcd9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70686186a30f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70686186a30f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7cef18db3479 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7cef18db3479 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6276585e3f3a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6276585e3f3a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71047ea1ce8c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71047ea1ce8c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a6d3b5446bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a6d3b5446bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d70f914affe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d70f914affe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc9161b89160 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc9161b89160 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e64196f8336 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e64196f8336 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 96ad3e984faa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 96ad3e984faa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd86b996d1da parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd86b996d1da parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 36c33d3f4f62 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 36c33d3f4f62 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0fb28e6bfca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0fb28e6bfca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 55dd9a488786 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 55dd9a488786 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec71daf7a7bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec71daf7a7bb 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195152 on provider checkpoint proc_lsst.multi INFO: cancelling 17195152 on provider checkpoint 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195092 on provider astro proc_lsst.multi INFO: cancelling 17195092 on provider astro 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195109 on provider astro proc_lsst.multi INFO: cancelling 17195109 on provider astro 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195134 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195134 on provider escience_a40 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195125 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195125 on provider escience_rtx6k 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195155 on provider checkpoint proc_lsst.multi INFO: cancelling 17195155 on provider checkpoint 2024-03-18 05:28:59 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro 2024-03-18 05:28:59 proc_lsst.multi:201 [INFO] cancelling 17195106 on provider astro proc_lsst.multi INFO: cancelling 17195106 on provider astro 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195142 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195142 on provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195153 on provider checkpoint proc_lsst.multi INFO: cancelling 17195153 on provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195154 on provider checkpoint proc_lsst.multi INFO: cancelling 17195154 on provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195151 on provider checkpoint proc_lsst.multi INFO: cancelling 17195151 on provider checkpoint 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195143 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195143 on provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195107 on provider astro proc_lsst.multi INFO: cancelling 17195107 on provider astro 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195108 on provider astro proc_lsst.multi INFO: cancelling 17195108 on provider astro 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195111 on provider astro proc_lsst.multi INFO: cancelling 17195111 on provider astro 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195130 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195130 on provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195147 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195147 on provider escience_a40 2024-03-18 05:29:00 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro 2024-03-18 05:29:00 proc_lsst.multi:201 [INFO] cancelling 17195113 on provider astro proc_lsst.multi INFO: cancelling 17195113 on provider astro 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195124 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195124 on provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195150 on provider checkpoint proc_lsst.multi INFO: cancelling 17195150 on provider checkpoint 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195128 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195128 on provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195144 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195144 on provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195126 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195126 on provider escience_rtx6k 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195094 on provider astro proc_lsst.multi INFO: cancelling 17195094 on provider astro 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195093 on provider astro proc_lsst.multi INFO: cancelling 17195093 on provider astro 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195145 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195145 on provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195132 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195132 on provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195146 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195146 on provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 2024-03-18 05:29:01 proc_lsst.multi:201 [INFO] cancelling 17195133 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195133 on provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195141 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195141 on provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 56975 on provider local proc_lsst.multi INFO: cancelling 56975 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 56975 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195138 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195138 on provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195104 on provider astro proc_lsst.multi INFO: cancelling 17195104 on provider astro 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195110 on provider astro proc_lsst.multi INFO: cancelling 17195110 on provider astro 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195127 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195127 on provider escience_rtx6k 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195131 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195131 on provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195099 on provider astro proc_lsst.multi INFO: cancelling 17195099 on provider astro 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195096 on provider astro proc_lsst.multi INFO: cancelling 17195096 on provider astro 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195098 on provider astro proc_lsst.multi INFO: cancelling 17195098 on provider astro 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195148 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195148 on provider escience_a40 2024-03-18 05:29:02 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro 2024-03-18 05:29:02 proc_lsst.multi:201 [INFO] cancelling 17195095 on provider astro proc_lsst.multi INFO: cancelling 17195095 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195149 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195149 on provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195139 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195139 on provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195122 on provider astro proc_lsst.multi INFO: cancelling 17195122 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195136 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195136 on provider escience_a40 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195123 on provider astro proc_lsst.multi INFO: cancelling 17195123 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195114 on provider astro proc_lsst.multi INFO: cancelling 17195114 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195101 on provider astro proc_lsst.multi INFO: cancelling 17195101 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195118 on provider astro proc_lsst.multi INFO: cancelling 17195118 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195100 on provider astro proc_lsst.multi INFO: cancelling 17195100 on provider astro 2024-03-18 05:29:03 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro 2024-03-18 05:29:03 proc_lsst.multi:201 [INFO] cancelling 17195115 on provider astro proc_lsst.multi INFO: cancelling 17195115 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195116 on provider astro proc_lsst.multi INFO: cancelling 17195116 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195137 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195137 on provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195140 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195140 on provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195117 on provider astro proc_lsst.multi INFO: cancelling 17195117 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195102 on provider astro proc_lsst.multi INFO: cancelling 17195102 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195112 on provider astro proc_lsst.multi INFO: cancelling 17195112 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195119 on provider astro proc_lsst.multi INFO: cancelling 17195119 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195135 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195135 on provider escience_a40 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195103 on provider astro proc_lsst.multi INFO: cancelling 17195103 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195097 on provider astro proc_lsst.multi INFO: cancelling 17195097 on provider astro 2024-03-18 05:29:04 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro 2024-03-18 05:29:04 proc_lsst.multi:201 [INFO] cancelling 17195120 on provider astro proc_lsst.multi INFO: cancelling 17195120 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-18 05:29:05 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-18 05:29:05 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17195150', '17195151', '17195152', '17195153', '17195154', '17195155', '17195092', '17195093', '17195094', '17195095', '17195096', '17195097', '17195098', '17195099', '17195100', '17195101', '17195102', '17195103', '17195104', '17195106', '17195107', '17195108', '17195109', '17195110', '17195111', '17195112', '17195113', '17195114', '17195115', '17195116', '17195117', '17195118', '17195119', '17195120', '17195122', '17195123', '17195130', '17195131', '17195132', '17195133', '17195134', '17195135', '17195136', '17195137', '17195138', '17195139', '17195140', '17195141', '17195142', '17195143', '17195144', '17195145', '17195146', '17195147', '17195148', '17195149', '17195124', '17195125', '17195126', '17195127', '17195128', '56975'] proc_lsst.multi INFO: new jobs since last cancel ['17195150', '17195151', '17195152', '17195153', '17195154', '17195155', '17195092', '17195093', '17195094', '17195095', '17195096', '17195097', '17195098', '17195099', '17195100', '17195101', '17195102', '17195103', '17195104', '17195106', '17195107', '17195108', '17195109', '17195110', '17195111', '17195112', '17195113', '17195114', '17195115', '17195116', '17195117', '17195118', '17195119', '17195120', '17195122', '17195123', '17195130', '17195131', '17195132', '17195133', '17195134', '17195135', '17195136', '17195137', '17195138', '17195139', '17195140', '17195141', '17195142', '17195143', '17195144', '17195145', '17195146', '17195147', '17195148', '17195149', '17195124', '17195125', '17195126', '17195127', '17195128', '56975'] 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195150 in provider checkpoint proc_lsst.multi INFO: found job 17195150 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195150 on provider checkpoint proc_lsst.multi INFO: cancelling 17195150 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195151 in provider checkpoint proc_lsst.multi INFO: found job 17195151 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195151 on provider checkpoint proc_lsst.multi INFO: cancelling 17195151 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195152 in provider checkpoint proc_lsst.multi INFO: found job 17195152 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195152 on provider checkpoint proc_lsst.multi INFO: cancelling 17195152 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195153 in provider checkpoint proc_lsst.multi INFO: found job 17195153 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195153 on provider checkpoint proc_lsst.multi INFO: cancelling 17195153 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195154 in provider checkpoint proc_lsst.multi INFO: found job 17195154 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195154 on provider checkpoint proc_lsst.multi INFO: cancelling 17195154 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195155 in provider checkpoint proc_lsst.multi INFO: found job 17195155 in provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195155 on provider checkpoint proc_lsst.multi INFO: cancelling 17195155 on provider checkpoint 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195092 in provider astro proc_lsst.multi INFO: found job 17195092 in provider astro 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195092 on provider astro proc_lsst.multi INFO: cancelling 17195092 on provider astro 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195093 in provider astro proc_lsst.multi INFO: found job 17195093 in provider astro 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195093 on provider astro proc_lsst.multi INFO: cancelling 17195093 on provider astro 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195094 in provider astro proc_lsst.multi INFO: found job 17195094 in provider astro 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195094 on provider astro proc_lsst.multi INFO: cancelling 17195094 on provider astro 2024-03-18 05:29:05 proc_lsst.multi:146 [INFO] found job 17195095 in provider astro proc_lsst.multi INFO: found job 17195095 in provider astro 2024-03-18 05:29:05 proc_lsst.multi:201 [INFO] cancelling 17195095 on provider astro proc_lsst.multi INFO: cancelling 17195095 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195096 in provider astro proc_lsst.multi INFO: found job 17195096 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195096 on provider astro proc_lsst.multi INFO: cancelling 17195096 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195097 in provider astro proc_lsst.multi INFO: found job 17195097 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195097 on provider astro proc_lsst.multi INFO: cancelling 17195097 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195098 in provider astro proc_lsst.multi INFO: found job 17195098 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195098 on provider astro proc_lsst.multi INFO: cancelling 17195098 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195099 in provider astro proc_lsst.multi INFO: found job 17195099 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195099 on provider astro proc_lsst.multi INFO: cancelling 17195099 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195100 in provider astro proc_lsst.multi INFO: found job 17195100 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195100 on provider astro proc_lsst.multi INFO: cancelling 17195100 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195101 in provider astro proc_lsst.multi INFO: found job 17195101 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195101 on provider astro proc_lsst.multi INFO: cancelling 17195101 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195102 in provider astro proc_lsst.multi INFO: found job 17195102 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195102 on provider astro proc_lsst.multi INFO: cancelling 17195102 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195103 in provider astro proc_lsst.multi INFO: found job 17195103 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195103 on provider astro proc_lsst.multi INFO: cancelling 17195103 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195104 in provider astro proc_lsst.multi INFO: found job 17195104 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195104 on provider astro proc_lsst.multi INFO: cancelling 17195104 on provider astro 2024-03-18 05:29:06 proc_lsst.multi:146 [INFO] found job 17195106 in provider astro proc_lsst.multi INFO: found job 17195106 in provider astro 2024-03-18 05:29:06 proc_lsst.multi:201 [INFO] cancelling 17195106 on provider astro proc_lsst.multi INFO: cancelling 17195106 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195107 in provider astro proc_lsst.multi INFO: found job 17195107 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195107 on provider astro proc_lsst.multi INFO: cancelling 17195107 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195108 in provider astro proc_lsst.multi INFO: found job 17195108 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195108 on provider astro proc_lsst.multi INFO: cancelling 17195108 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195109 in provider astro proc_lsst.multi INFO: found job 17195109 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195109 on provider astro proc_lsst.multi INFO: cancelling 17195109 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195110 in provider astro proc_lsst.multi INFO: found job 17195110 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195110 on provider astro proc_lsst.multi INFO: cancelling 17195110 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195111 in provider astro proc_lsst.multi INFO: found job 17195111 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195111 on provider astro proc_lsst.multi INFO: cancelling 17195111 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195112 in provider astro proc_lsst.multi INFO: found job 17195112 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195112 on provider astro proc_lsst.multi INFO: cancelling 17195112 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195113 in provider astro proc_lsst.multi INFO: found job 17195113 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195113 on provider astro proc_lsst.multi INFO: cancelling 17195113 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195114 in provider astro proc_lsst.multi INFO: found job 17195114 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195114 on provider astro proc_lsst.multi INFO: cancelling 17195114 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195115 in provider astro proc_lsst.multi INFO: found job 17195115 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195115 on provider astro proc_lsst.multi INFO: cancelling 17195115 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195116 in provider astro proc_lsst.multi INFO: found job 17195116 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195116 on provider astro proc_lsst.multi INFO: cancelling 17195116 on provider astro 2024-03-18 05:29:07 proc_lsst.multi:146 [INFO] found job 17195117 in provider astro proc_lsst.multi INFO: found job 17195117 in provider astro 2024-03-18 05:29:07 proc_lsst.multi:201 [INFO] cancelling 17195117 on provider astro proc_lsst.multi INFO: cancelling 17195117 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195118 in provider astro proc_lsst.multi INFO: found job 17195118 in provider astro 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195118 on provider astro proc_lsst.multi INFO: cancelling 17195118 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195119 in provider astro proc_lsst.multi INFO: found job 17195119 in provider astro 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195119 on provider astro proc_lsst.multi INFO: cancelling 17195119 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195120 in provider astro proc_lsst.multi INFO: found job 17195120 in provider astro 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195120 on provider astro proc_lsst.multi INFO: cancelling 17195120 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195122 in provider astro proc_lsst.multi INFO: found job 17195122 in provider astro 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195122 on provider astro proc_lsst.multi INFO: cancelling 17195122 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195123 in provider astro proc_lsst.multi INFO: found job 17195123 in provider astro 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195123 on provider astro proc_lsst.multi INFO: cancelling 17195123 on provider astro 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195130 in provider escience_a40 proc_lsst.multi INFO: found job 17195130 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195130 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195130 on provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195131 in provider escience_a40 proc_lsst.multi INFO: found job 17195131 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195131 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195131 on provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195132 in provider escience_a40 proc_lsst.multi INFO: found job 17195132 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195132 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195132 on provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195133 in provider escience_a40 proc_lsst.multi INFO: found job 17195133 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195133 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195133 on provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195134 in provider escience_a40 proc_lsst.multi INFO: found job 17195134 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195134 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195134 on provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:146 [INFO] found job 17195135 in provider escience_a40 proc_lsst.multi INFO: found job 17195135 in provider escience_a40 2024-03-18 05:29:08 proc_lsst.multi:201 [INFO] cancelling 17195135 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195135 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195136 in provider escience_a40 proc_lsst.multi INFO: found job 17195136 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195136 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195136 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195137 in provider escience_a40 proc_lsst.multi INFO: found job 17195137 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195137 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195137 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195138 in provider escience_a40 proc_lsst.multi INFO: found job 17195138 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195138 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195138 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195139 in provider escience_a40 proc_lsst.multi INFO: found job 17195139 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195139 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195139 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195140 in provider escience_a40 proc_lsst.multi INFO: found job 17195140 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195140 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195140 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195141 in provider escience_a40 proc_lsst.multi INFO: found job 17195141 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195141 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195141 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195142 in provider escience_a40 proc_lsst.multi INFO: found job 17195142 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195142 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195142 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195143 in provider escience_a40 proc_lsst.multi INFO: found job 17195143 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195143 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195143 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195144 in provider escience_a40 proc_lsst.multi INFO: found job 17195144 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195144 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195144 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195145 in provider escience_a40 proc_lsst.multi INFO: found job 17195145 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195145 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195145 on provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:146 [INFO] found job 17195146 in provider escience_a40 proc_lsst.multi INFO: found job 17195146 in provider escience_a40 2024-03-18 05:29:09 proc_lsst.multi:201 [INFO] cancelling 17195146 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195146 on provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195147 in provider escience_a40 proc_lsst.multi INFO: found job 17195147 in provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195147 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195147 on provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195148 in provider escience_a40 proc_lsst.multi INFO: found job 17195148 in provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195148 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195148 on provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195149 in provider escience_a40 proc_lsst.multi INFO: found job 17195149 in provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195149 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195149 on provider escience_a40 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195124 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195124 in provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195124 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195124 on provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195125 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195125 in provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195125 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195125 on provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195126 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195126 in provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195126 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195126 on provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195127 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195127 in provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195127 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195127 on provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 17195128 in provider escience_rtx6k proc_lsst.multi INFO: found job 17195128 in provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 17195128 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17195128 on provider escience_rtx6k 2024-03-18 05:29:10 proc_lsst.multi:146 [INFO] found job 56975 in provider local proc_lsst.multi INFO: found job 56975 in provider local 2024-03-18 05:29:10 proc_lsst.multi:201 [INFO] cancelling 56975 on provider local proc_lsst.multi INFO: cancelling 56975 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 56975 parsl.providers.local.local WARNING: Failed to kill PID: 56975 and child processes on local 2024-03-18 05:29:11 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 534.8467 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20211003_B1b_science#step1_20240318T051950Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 534.8527 seconds; current memory usage: 0.379 Gibyte, delta: 0.007 Gibyte, peak delta: 0.007 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 554.8694 seconds; current memory usage: 0.379 Gibyte, delta: 0.240 Gibyte, peak delta: 0.240 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.379 Gibyte (main), 0.379 Gibyte (largest child process) Run Id: None Run Name: DEEP_20211003_B1b_science#step1_20240318T051950Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 9m17.152s user 1m40.297s sys 0m53.137s